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Analysis  of  Baseline  Measures  in  the  Medicare  Current 
Beneficiary  Survey  for  Use  in  Monitoring  the  National  Medicare 

Education  Program 

Executive  Summary 


Background 

Current  changes  to  the  Medicare  program  increase  the  number  and  range  of 
choices  available  to  beneficiaries.  The  creation  of  Medicare+Choice  and  Medicare 
HMOs  as  well  as  changes  within  the  fee-for-service  program  over  time  have  presented 
beneficiaries  with  many  challenges  in  understanding  their  benefits  and  options.  The 
Health  Care  Financing  Administration  (HCFA)  and  the  Congress  have  expressed  concern 
about  the  ability  of  Medicare  beneficiaries  to  make  informed  choices  in  this  changing 
environment. 

Recent  research  has  documented  the  low  level  of  understanding  that  beneficiaries 
possess  about  the  Medicare  program.  Many  beneficiaries  do  not  understand  what  is 
covered,  who  operates  the  program,  or  what  plan  options  are  available  to  them.  Many 
have  never  heard  of  a  Medicare  HMO  and  the  vast  majority  is  unable  to  identify  the 
differences  between  the  original  Medicare  program  and  a  Medicare  managed  care  plan. 
This  is  an  especially  important  finding,  as  understanding  the  differences  among  original 
Medicare,  Medicare  HMOs,  and  the  newer  Medicare+Choice  options  is  a  prerequisite  to 
making  an  informed  choice.  Thus,  the  need  is  greater  than  ever  for  easily  understood 
information  about  Medicare  that  beneficiaries  are  motivated  to  use. 

The  National  Medicare  Education  Program  (NMEP)  is  HCFA's  coordinated  effort 
to  address  this  apparent  lack  of  beneficiary  understanding,  in  both  the  short  and  long 
term,  by  creating  useful  information  resources  (e.g.,  print  materials,  Internet  materials, 
toll-free  information  hotlines,  trained  intermediaries),  informing  beneficiaries  that  these 
resources  are  available,  and  motivating  them  to  use  the  resources. 

The  goals  of  the  NMEP  are: 

■  increasing  beneficiary  access  to  information, 

-  raising  beneficiary  awareness  of  these  new  information  sources, 

-  helping  beneficiaries  understand  the  choices  that  can  be  made,  and 

■  helping  beneficiaries  use  the  information  obtained  to  make  an  optimal  choice. 

HCFA  must  identify  and  use  data  collected  from  beneficiaries  to  monitor  progress  toward 
achievement  of  the  NMEP  goals,  and  measuring  progress  using  knowledge  constructs  or 
indices  is  an  important  avenue  for  determining  if  the  NMEP  is  successful. 
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The  purpose  of  this  report  is  to  recommend  Medicare  Current  Beneficiary  Survey 
(MCBS)  variables  that  provide  a  basehne  for  measuring  progress  toward  achievement  of 
the  four  NMEP  goals,  and  to  help  HCFA  identify  effective  measures  of  beneficiary 
knowledge  that  can  be  used  to  evaluate  the  NMEP.  This  effort  will  be  achieved  by 
accomplishing  the  following  four  objectives: 

■  Identify  measures  in  the  MCBS  that  potentially  may  be  used  as  baseline 
measures; 

■  Evaluate  the  reliability  and  validity  of  potential  baseline  MCBS  variables  and 
recommend  measures  to  be  used  for  measuring  NMEP  progress; 

■  Assess  how  these  recommended  measures  vary  with  important  beneficiary 
attributes;  and 

■  Identify  important  knowledge  measures  from  prior  work  on  the  NMEP  that 
can  be  used  in  future  monitoring  of  the  NMEP. 

Two  reports  are  planned  for  this  project.  This  Phase  1  report  will  identify  MCBS 
variables  that  can  be  used  to  establish  a  pre-NMEP  baseline.  The  Phase  2  report  will 
identify  progress  toward  NMEP  goals  using  1998  MCBS  Rounds  23  and  24,  which 
immediately  follow  NMEP  implementation.  The  Phase  2  report  contains  analyses  using 
MCBS  variables  that  were  explicitly  created  to  measure  progress  toward  NMEP  goals, 
and  can  be  used  in  future  longitudinal  analyses. 

Key  Findings 

Four  measures  related  to  knowledge  were  identified  in  the  MCBS  that  may 
potentially  serve  as  baseline  measures  for  monitoring  the  NMEP.  One  measure 
concemed  beneficiaries'  perceived  understanding  of  Medicare  generally,  two  measures 
concerned  their  knowledge  of  benefits,  and  one  measure  concemed  their  perceived 
knowledge  about  specific  aspects  of  the  Medicare  program.  Of  these  four  measures,  only 
the  4-item  quiz  in  Round  1 8  and  the  Know  All  Need  To  Know  Index  in  Rounds  1 8  and 
24  were  found  to  be  acceptable  with  respect  to  internal  consistency  reliability  and 
construct  validity. 

To  investigate  the  effects  of  the  NMEP  on  beneficiary  knowledge  as  measured  by 
these  two  indexes,  descriptive  analyses  were  conducted.  Persons  who  were  hypothesized 
to  be  more  knowledgeable  generally  had  the  most  favorable  knowledge  outcomes. 
Persons  who  were  married,  male,  65-75  years  of  age,  white  non-Hispanic,  had  more  than 
$25,000  in  income,  in  households  with  two  persons,  and  college  educated  were  more 
knowledgeable.  In  addition,  persons  who  did  not  have  a  proxy  answer  for  them,  had  no 
limitations  in  the  activities  of  daily  living,  and  reported  themselves  to  be  in  very  good  or 
excellent  health  status  were  more  knowledgeable.  Finally,  persons  without  disabilities, 
who  had  Medicare  and  private  insurance,  and  with  some  utilization  of  health  services 
were  more  knowledgeable. 

In  general,  persons  who  were:  divorced,  separated,  or  otherwise  living  alone; 
under  age  65;  had  minority  group  status;  had  lower  income;  lived  in  larger  households; 
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had  less  than  an  8'*'  grade  education;  had  several  limitations  in  the  activities  of  daily 
living;  said  they  were  in  fair  or  poor  health,  were  disabled;  had  both  Medicare  and 
Medicaid;  and  had  no  use  of  services  were  less  knowledgeable.  These  persons  may  be 
the  most  appropriate  persons  toward  whom  to  target  knowledge  improvement  efforts.  The 
NMEP  strategy  may  prove  more  efficient  if  focused  on  characteristics  where  there  was  a 
large  variance  in  the  distribution  of  responses,  and  then  targeted  to  those  individuals  who 
scored  much  lower  than  other  respondents.  An  example  of  a  characteristic  with  a  large 
variance  is  Type  of  Health  Insurance,  where  persons  with  both  Medicare  and  Medicaid 
scored  substantially  lower  on  knowledge  measures  than  persons  in  other  insurance 
categories. 

RTI  is  conducting  two  other  studies  for  HCFA  in  which  it  has  created  and 
evaluated  indices  to  measure  knowledge  in  the  Medicare  population.  These  studies 
include  the  Evaluation  of  Medicare  CAHPS  and  the  Medicare  &  You  1999  Handbook  in 
Kansas  City  and  the  Expanded  (National)  Evaluation  of  the  Medicare  &  You  2000 
Handbook.  For  each  of  these  evaluations,  RTI  created  a  knowledge  index  to  assess  how 
well  beneficiaries  understand  the  Medicare  program  and  related  health  insurance  options. 

We  compared  the  psychometric  performance  of  these  two  indices  and  the  8-item 
quiz  from  MCBS  Round  23  to  determine  which  provides  the  best  measure  of  knowledge 
for  future  research.  Based  on  the  results  of  the  scale-level  and  item-level  psychometric 
analyses,  the  National  Evaluation  index  performed  best.  This  index  had  the  highest 
reliability  of  the  three  scales  and  demonstrated  evidence  of  construct  validity.  The  index 
contains  a  variety  of  difficulty  levels,  allowing  it  to  measure  the  knowledge  of 
beneficiaries  with  a  wide  range  of  ability  levels.  Finally,  the  National  Evaluation  index 
performed  well  with  respect  to  item  content.  Results  of  the  factor  analyses  as  well  as  the 
content  of  the  items  suggest  that  this  scale  measures  a  wider  range  of  topics  in 
comparison  to  the  other  two  knowledge  scales.  This  diversity  allows  the  scale  to  more 
accurately  assess  the  knowledge  beneficiaries  need  to  successfully  navigate  the  Medicare 
program. 

We  also  compared  questions  that  were  common  to  both  the  MCBS  and  the 
National  Evaluation.  One  set  of  questions  dealt  with  the  use  of  the  handbook  by 
beneficiaries  and  the  second  set  were  knowledge  quizzes.  Only  two  questions  concerning 
the  use  of  the  handbook  could  be  compared  between  the  two  surveys.  National 
Evaluation  respondents  were  more  likely  to  read  and  keep  the  handbook  than  MCBS 
respondents.  Five  of  the  quiz  questions  could  be  compared  directly.  On  three  of  the  five 
questions  there  was  considerable  difference  in  responses,  with  respondents  from  the 
National  Evaluation  scoring  higher  on  two  of  the  three  questions,  perhaps  because  these 
persons  had  received  the  handbook  as  an  intervention  and  knew  they  were  being 
evaluated. 
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Conclusion 

Two  sets  of  MCBS  questions  form  knowledge  indices  that  may  be  used  as 
baseline  measures  in  tracking  the  progress  of  the  NMEP.  One  index  measures 
beneficiary  knowledge  using  4  questions  with  objective  answers,  and  the  other  index 
measures  the  perceived  knowledge  of  beneficiaries  using  5  questions.  In  general,  persons 
who  were  socio-economically  disadvantaged,  had  health  limitations,  had  Medicaid,  and 
who  did  not  use  any  services  scored  lower  on  these  knowledge  indices  in  the  1996 
MCBS.  However,  questions  from  the  Expanded  (National)  Evaluation  of  the  Medicare 
&  You  2000  Handbook  performed  better  than  the  MCBS  questions. 

Future  efforts  are  needed  to  develop  an  analysis  framework  for  MCBS 
longitudinal  modeling  to  track  NMEP  progress,  develop  additional  measures  for 
assessing  the  impact  of  the  NMEP,  make  direct  comparisons  among  knowledge  questions 
administered  to  the  same  population,  and  create  an  optimum  knowledge  index  using  item 
response  theory,  which  enables  the  knowledge  scale  to  vary  in  response  to  changes  in 
NMEP  objectives,  while  maintaining  the  ability  to  measure  trends. 
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1.0  Introduction 


1.1     Background  on  NMEP 

Recent  changes  to  the  Medicare  program  increase  the  number  and  range  of  options  open 
to  individual  beneficiary  choice.  Understanding  these  options  and  choosing  optimally  places  a 
greater  burden  on  beneficiaries  at  the  same  time  that  they  express  concern  over  decreasing 
cognitive  acuity  from  aging  (Gibbs  et  al.,  1996).  Thus,  the  need  is  greater  than  ever  for  easily 
understood  information  about  Medicare  that  beneficiaries  are  motivated  to  use. 

Several  studies  have  documented  that  adults  of  all  ages  have  an  inadequate  understanding 
of  their  health  insurance  coverage  (Mechanic,  1989;  Isaacs,  1996;  Gamick,  1993).  Now  there  is 
a  growing  body  of  research  documenting  the  low  level  of  understanding  that  Medicare 
beneficiaries  possess  about  the  Medicare  program  (Gibbs  et  al.,  1996;  Hibbard  et  al.,  1998; 
Murray  and  Shatto,  998;  National  Academy  of  Social  Insurance  (NASI),  1998).  Many 
beneficiaries  do  not  understand  what  is  covered,  who  operates  the  program,  or  what  plan  options 
are  available  under  the  Medicare  program.  Many  have  never  heard  of  a  Medicare  HMO  and  the 
vast  majority  are  unable  to  identify  the  differences  between  the  original  Medicare  program  and  a 
Medicare  managed  care  plan  (Hibbard  et  al.,  1998).  This  is  an  especially  important  finding,  as 
understanding  the  differences  among  original  Medicare,  Medicare  HMOs,  and  the  newer 
Medicare+Choice  options  is  a  prerequisite  to  making  an  informed  choice.  The  National 
Medicare  Education  Program  (NMEP)  is  the  Health  Care  Financing  Administration's  (HCFA) 
coordinated  effort  to  address  this  problem,  in  both  the  short  and  long  term,  by  creating  useful 
information  resources  (e.g.,  print  materials,  Internet  materials,  toll-free  information  hotlines, 
trained  intermediaries),  informing  beneficiaries  that  these  resources  are  available,  and  motivating 
them  to  use  the  resources. 

The  goals  of  the  NMEP  are: 

increasing  beneficiary  access  to  information, 

•  raising  beneficiary  awareness  of  these  new  information  sources, 

•  helping  beneficiaries  understand  the  choices  that  can  be  made,  and 

-    helping  beneficiaries  use  the  information  obtained  to  make  an  optimal  choice. 

While  beneficiary  knowledge  per  se  is  not  an  NMEP  goal,  it  is  an  important  construct  for 
measuring  attainment  of  the  NMEP  goals.  HCFA  must  identify  and  use  data  collected  from 
beneficiaries  to  monitor  progress  toward  the  NMEP  goals,  and  measuring  progress  using 
knowledge  constructs  or  indices  is  one  important  avenue  for  determining  if  the  NMEP  is 
successful. 
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1.2  Purpose 


The  purpose  of  this  report  is  to  recommend  Medicare  Current  Beneficiary  Survey 
(MCBS)  variables  that  provide  a  baseline  for  measuring  progress  toward  achievement  of  the  four 
NMEP  goals,  and  to  help  HCFA  identify  effective  measures  of  beneficiary  knowledge.  This 
effort  involves  the  following  objectives: 

■  Identify  items  in  the  MCBS  that  may  potentially  be  used  as  baseline  measures; 
«    Evaluate  the  reliability  and  validity  of  potential  baseline  MCBS  variables  for 

measuring  NMEP  progress; 

■  Assess  how  these  measures  vary  with  important  beneficiary  attributes;  and 

•    Identify  important  knowledge  measures  from  prior  work  on  the  NMEP  that  can  be 
used  in  future  monitoring  of  the  NMEP. 

This  report  is  the  first  of  two  reports  concerning  how  the  MCBS  can  be  used  to 
effectively  measure  progress  toward  NMEP  goals.  While  this  Phase  1  report  will  identify  MCBS 
variables  that  could  be  used  to  establish  a  pre-NMEP  baseline,  the  Phase  2  report  will  identify 
progress  toward  NMEP  goals  using  1998  MCBS  Rounds  23  and  24  which  immediately  followed 
full  national  NMEP  implementation.  The  Phase  2  report  contains  analyses  using  MCBS 
variables  that  were  explicitly  created  to  measure  progress  toward  achievement  of  NMEP  goals, 
and  potentially  can  be  used  in  future  longitudinal  analyses. 

Two  companion  reports  accompany  the  Phase  1  and  Phase  2  reports.  The  Technical  Note 
1:  Knowledge  Index  contains  psychometric  analyses  of  four  potential  measures  that  we 
considered  for  measuring  NMEP  progress.  Technical  Note  2:  Preliminary  Baseline  Analysis 
contains  the  foundation  for  longitudinal  analyses  that  takes  advantage  of  the  5-state  pilot  project 
where  the  Medicare  &  You  handbook  was  initially  fielded.  All  of  these  reports  serve  to  provide 
HCFA  with  information  and  recommendations  for  using  the  MCBS  to  measure  progress  toward 
achievement  of  the  NMEP  goals. 

2.0  Potential  MCBS  Baseline  Measures  for  Monitoring  the  NMEP 

2.1  Rationale  for  Using  Knowledge  to  Evaluate  Achievement  of  NMEP  Goals 

Progress  toward  accomplishment  of  the  NMEP  goals  is  best  determined  by  measuring 
beneficiaries'  knowledge  of  the  Medicare  program.  Ideally,  measures  should  be  found  to  track 
progress  toward  each  of  the  goals  of  the  NMEP.  Items  that  precede  the  implementation  of  the 
NMEP  would  be  likely  candidates  for  monitoring  progress  toward  each  of  these  NMEP  goals  as 
they  could  be  used  in  a  longitudinal  pretest  and  post  test  design.  A  model  for  using  knowledge 
to  evaluate  progress  toward  achievement  of  NMEP  goals  is  displayed  in  Exhibit  1. 

Exhibit  1 .      Logic  Model  for  Evaluating  Achievement  of  NMEP  Goals 


Lack  of  Knowledge     NMEP  Activities  ^  Increased  Knowledge  ->  Desired  NMEP  Outcomes 
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We  constructed  potential  measures  of  knowledge  from  individual  MCBS  questions  to 
develop  single  or  multi-item  knowledge  indices  designed  to  measure  achievement  of  NMEP 
goals.  We  then  analyzed  data  for  each  of  the  knowledge  indices  constructed  to  provide  estimates 
of  their  internal  consistency  (reliability)  and  construct  validity.  These  two  characteristics  are 
important  in  determining  whether  scores  yielded  by  a  knowledge  index  are  repeatable  and 
whether  they  provide  important  information  about  beneficiary  knowledge.  We  then  used  the 
results  of  these  psychometric  analyses  to  make  recommendations  of  which  knowledge  indices  to 
use  as  MCBS  baseline  measures  for  monitoring  the  achievement  of  NMEP  goals. 

2.2     Candidates  for  Baseline  IVIeasures 

Only  four  measures  appear  in  both  pre-  and  post-NMEP  rounds  of  the  MCBS  and  can  be 
considered  for  evaluation  as  potential  baseline  knowledge  measures.  Other  measures  that  we 
considered  as  potential  baseline  measures  focus  on  a  need  for  information  or  satisfaction.  While 
these  measures  may  provide  information  that  could  be  used  for  other  purposes,  they  are  not 
recommended  as  knowledge  measures  by  which  one  could  monitor  the  effects  of  NMEP 
activities.  The  four  measures  we  used  are: 

(1)  perceived  understanding  of  Medicare.  A  single  yes  or  no  dichotomous  question  that 
asks  whether  Medicare  is  understandable,  but  only  in  the  1995  to  1997  MCBS.  This 
is  a  global  indicator  of  understanding  of  the  Medicare  program,  but  it  does  not 
continue  past  1997,  and  its  presumed  "replacement"  in  the  1998  and  1999  MCBS  is 
quite  a  bit  different. 

(2)  Medicare  benefits  knowledge  in  1996.  A  knowledge  index  about  benefits  covered  by 
Medicare  found  in  the  1996  MCBS.  The  index  is  a  true-false  quiz  composed  of  four 
questions.  This  quiz  is  limited  in  its  use  as  a  knowledge  index  to  measure  NMEP 
progress  over  time  as  it  has  so  few  questions.  This  measure  of  program  knowledge  is 
also  limited  to  benefits  available  in  the  Medicare  program  and  does  not  include 
questions  about  Medicare  options  and  choices. 

(3)  Medicare  benefits  knowledge  in  1998.  A  knowledge  index  about  benefits  covered  by 
Medicare  found  in  the  1998  MCBS.  This  index  is  also  a  true-false  quiz  composed  of 
three  questions,  only  one  of  which  is  the  same  as  the  four-item  quiz  in  the  1996 
MCBS.  This  quiz  is  also  limited  in  its  use  as  a  knowledge  index  as  it  has  even  fewer 
questions  and  focuses  on  benefits  and  not  on  options  or  choices  of  beneficiaries.  The 
1998  MCBS  items  have  been  repeated  in  the  1999  MCBS. 
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(4)  perceived  knowledge  of  aspects  of  Medicare.  A  knowledge  measure  derived  from 
five  out  of  a  series  of  seven  questions  (one  item  that  applies  only  to  Medicaid 
recipients  and  another  that  refers  to  staying  healthy  were  not  included  in  the  index), 
each  with  5  levels  of  response.  This  knowledge  measure  reflects  how  much 
beneficiaries  perceive  they  know  about  particular  aspects  of  the  Medicare  program. 
Each  question  in  the  index  appeared  in  the  1996,  1998  and  1999  MCBS.  It  does  not 
measure  "knowledge"  in  a  direct  manner  but  attempts  to  capture  how  much  the 
beneficiaries  think  they  know  about  multiple  aspects  of  the  Medicare  program.  One 
would  expect  the  NMEP  to  increase  the  level  of  perceived  knowledge  of  the  program 
over  time.  Measured  knowledge  is  likely  to  be  moderately  related  to  perceived 
knowledge. 

Changes  in  these  measures  over  time  could  serve  as  outcomes  that  are  used  to  evaluate 
progress  toward  achievement  of  NMEP  goals.  In  the  following  section  the  ability  of  each  of 
these  measures  to  effectively  monitor  the  NMEP  is  evaluated.  The  items  used  to  create  the 
indices  are  presented  in  Exhibit  2. 


Exhibit  2.  Items  from  the  1995-1998  MCBS  Used  to  Construct  Knowledge  Indices  to 
Serve  as  Baseline  Measures  to  Monitor  Impact  of  NMEP 


1 

Perceived  understanding  of  Medicare  (Found  in  the  1995, 1996  and  1997  MCBS  in  the  PR 
section  of  rounds  14, 17  and  20.) 

(1) 

In  general,  do  you  think  the  Medicare  program  is  understandable? 

Medicare  Benefits  Knowledge  (Found  in  the  1996  MCBS  in  the  OL  section  of  round  18.) 

(1) 

Medicare  pays  for  flu  shots. 

(2) 

Medicare  pays  for  an  annual  physical  exam. 

(3) 

A  doctor  who  accepts  assignment  can't  charge  more  than  Medicare  allows  for  covered  services. 

(4) 

If  you  do  not  agree  with  a  decision  Medicare  makes  on  a  claim  from  a  hospital  or  doctor,  such 
as  whether  it  will  cover  the  service  or  how  much  it  will  pay,  you  can  appeal  the  decision. 

Medicare  Benefits  Knowledge  (Found  in  the  1998  MCBS  in  the  BN  section  of  round  24.) 

(1) 

Medicare  covers  colorectal  cancer  screening. 

(2) 

Medigap  or  supplemental  insurance  is  the  same  as  a  Medicare  managed  care  plan. 

(3) 

Medicare  covers  an  annual  flu  shot. 

4           Perceived  Knowledge  of  aspects  of  Medicare  (Found  in  the  1996, 1998  and  1999  MCBS  in 
the  OL/BN  section  of  rounds  18, 24  and  27.) 

(1) 

How  much  do  you  feel  you  know  about  the  Medicare  program,  such  as  what  services  Medicare 
covers  or  does  not  cover? 

(2) 

How  much  do  you  feel  you  know  about  how  much  you  have  to  pay  for  medical  services 
covered  by  Medicare? 

(3) 

How  much  do  you  feel  you  know  about  supplemental  or  Medigap  insurance,  such  as  what  it 
covers  or  how  it  works  with  Medicare  to  pay  medical  claims? 

(4) 

How  much  do  you  feel  you  know  about  the  availability  and  benefits  of  Medicare  Health 
Maintenance  Organizations? 

(5) 

How  much  do  you  feel  you  know  about  choosing  or  finding  a  doctor  or  other  healthcare 
provider? 
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3.0  Index  Construction  and  Psychometric  Testing  of  Candidate 
IVIeasures 

In  this  section  we  describe  how  each  of  the  knowledge  indices  was  constructed,  and 
provide  a  summary  of  the  results  of  psychometric  tests  conducted.  Based  on  these  tests,  we 
recommend  baseline  knowledge  measures  drawn  from  MCBS  questions  that  can  be  used  for 
evaluating  the  achievement  of  NMEP  goals. 

3.1  Scoring  Algorithms 

Knowledge  indices  were  created  for  each  series  of  MCBS  questions  described  above.  A 
knowledge  index  aggregates  responses  to  individual  questions  in  each  set,  resulting  in  a  metric 
used  for  recording  a  single  score  across  all  questions  in  the  set.  A  total  of  four  different 
knowledge  indices  were  created.  The  first  index  measures  perceived  understanding  of  the 
Medicare  program  '  i  general  and  appears  only  in  1996.  The  second  and  third  indices  are  drawn 
from  questions  about  Medicare  benefits,  with  four  questions  in  the  index  for  the  1996  MCBS, 
and  three  questions  for  the  index  in  the  1998  MCBS.  The  fourth  index  measures  perceived 
knowledge  of  specific  aspects  of  the  Medicare  program.  This  index  is  the  same  in  the  1996  and 
the  1998  MCBS.  The  construction  of  each  of  these  as  knowledge  indices  is  discussed  more  fully 
in  the  following  sections. 

Medicare  Understandability  Question 

This  knowledge  index  is  a  single  question  from  the  PR  sections  of  the  1995  through  1997 
MCBS  and  asks  whether  respondents  perceive  that  they  understand  the  Medicare  program  as  a 
whole.  Affirmative  ("Yes")  responses  to  this  quesfion  were  coded  as  "1,"  while  responses  of 
"No"  were  coded  as  "0."  Responses  of  "Don't  Know,"  "Refused,"  and  "Not  Ascertained"  were 
coded  as  missing. 

Four-Item  Quiz 

In  the  1996  MCBS  Round  18,  four  questions  were  asked  of  respondents  in  the  form  of  a 
quiz  on  Medicare  benefits  and  processes.  Correct  responses  to  each  of  the  four  quiz  questions 
were  coded  as  "1,"  while  all  other  responses  (incorrect  responses  and  don't  know)  were  coded  as 
"0."  A  fifth  potential  quiz  question  addressing  Medicare  coverage  for  mammograms  was 
excluded  from  the  score  calculation  because  it  was  only  asked  of  women.  The  recorded 
responses  were  then  summed  to  create  the  quiz  scores.  For  example,  the  score  for  a  respondent 
who  answered  two  out  of  the  four  quiz  questions  correctly  would  be  calculated  as 

0+1  +  1  +  0  =  2. 

An  advantage  of  this  scoring  method  is  that  it  produces  scores  that  have  a  meaningful 
interpretation,  specifically,  the  number  of  questions  for  which  the  respondent  knew  the  correct 
answer.  Scores  range  from  0  to  4,  with  higher  quiz  scores  reflecting  more  correct  responses 
about  Medicare  benefits  coverage. 
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Three-Item  Quiz 

Similar  scoring  methods  were  used  for  a  three-item  quiz  concerning  Medicare  benefits  in 
the  1998  MCBS  Round  24.  As  with  the  four-item  quiz,  a  potential  quiz  question  concerning 
Medicare  coverage  of  mammograms  was  excluded  because  it  was  asked  only  of  women.  The 
responses  to  the  three  remaining  quiz  questions  were  recoded  so  that  correct  responses  were 
coded  as  "1"  and  all  other  responses  (incorrect  responses  and  don't  know)  were  coded  as  "0." 
The  quiz  score  was  computed  by  summing  the  recoded  responses  to  the  three  quiz  questions. 
This  produces  a  score  representing  the  total  number  of  questions  answered  correctly.  For 
example,  a  respondent  who  had  only  one  correct  response  and  two  incorrect  responses  received  a 
score  of  1  as  follows: 

l-hO  +  0=  1 

This  scoring  system  produces  scores  ranging  from  0  to  3  with  higher  scores  indicating  more 
knowledge  of  the  Medicare  system. 

Know-AII-Need-to-Know  Index 

This  knowledge  index  is  calculated  from  round  18  of  the  1996  MCBS  and  represents  a 
respondent's  perceived  knowledge  about  whether  they  think  they  know  all  they  need  to  know 
about  certain  aspects  of  the  Medicare  program.  The  items  in  this  index  were  also  included  in  the 
1998  and  1999  MCBS.  The  know-all-need-to-know  index  was  created  by  reverse  scoring  each 
of  the  five  response  categories  across  the  questions.  For  example,  in  the  original  coding  of  the 
variables,  knowing  "Just  about  everything  I  need  to  know..."  was  coded  as  "1"  and  knowing 
"Almost  none  of  what  I  need  to  know. . ."  was  coded  as  "5."  The  former  response  was  recoded 
as  "5"  while  the  latter  was  recoded  as  "1 ."  Responses  for  "2"  and  "4"  were  also  switched. 
Responses  were  then  summed. 

If  a  respondent's  recoded  answers  to  the  original  questions  were  "3,"  "4,"  "2,"  "1,"  and 
"5",  the  knowledge  score  would  be  calculated  as  follows: 

3+4  +  2+1+5  =15. 

Higher  scores  on  this  index  reflect  beneficiaries  reporting  that  they  knew  more  of  what  they 
thought  they  needed  to  know  across  five  different  topics.  This  scoring  system  produces  scores 
that  may  theoretically  range  from  5  to  25,  thus  providing  greater  variability  in  scores  and  more 
power  to  discriminate  among  and  within  beneficiaries. 

Where  data  were  missing,  the  know-all-need-to-know  index  was  calculated  by  imputing 
values  rather  than  altogether  eliminating  a  respondent's  answers.  Imputation  was  only  used  for 
respondents  who  answered  at  least  three  of  the  know-all-need-to-know  items  by  substituting  the 
mean  value  of  the  answered  items  for  the  missing  items  (Chapman,  1976).  Individuals  with 
missing  responses  for  more  than  two  of  the  know-all-need-to-know  questions  were  assigned  a 
value  of  missing  for  this  index. 


6 


3.2     Psychometric  Testing  of  Knowledge  Indices 

Psychometric  analyses  were  conducted  on  the  knowledge  indices  described  above  to 
determine  their  reliability  and  validity  as  measures  of  knowledge.  (The  Knowledge  Index 
Technical  Note  dated  May  5,  2000  contains  a  flill  report  of  all  psychometric  testing  conducted  on 
these  and  other  knowledge  indices  and  serves  as  a  companion  deliverable  to  this  report.)  The 
reliability  of  the  scales  was  assessed  using  internal  consistency.  Internal  consistency  measures 
the  degree  to  which  items  on  a  scale  are  related  to  each  other  and  therefore  appear  to  be 
measuring  the  same  construct.  The  internal  consistency  reliability  of  all  multi-item  indices  was 
estimated  using  Cronbach's  alpha  coefficient  (Cronbach,  1951).  Coefficient  alpha  was  not 
calculated  for  the  scale  measuring  perceived  understanding  of  Medicare  because  it  contains  only 
one  item.  Test-retest  reliability  of  the  knowledge  indexes  was  not  conducted  as  the  time  period 
between  responses  was  approximately  two  years,  and  test-retest  reliability  is  usually  measured 
over  much  shorter  intervals. 

Construct  validity  was  assessed  by  determining  if  scale  scores  discriminated  among 
groups  of  Medicare  beneficiaries  who  would  be  expected  to  differ  in  their  knowledge  of 
Medicare.  Based  on  previous  research  (McCall,  Rice  and  Sangl,  1986;  Hibbard,  Jewett, 
Englemann,  and  Tusler,  1998;  McCormack,  Garfinkel,  Hibbard,  Keller,  Kilpatrick,  and  Kosiak, 
2000),  it  was  expected  that  the  following  groups  of  beneficiaries  would  have  higher  levels  of 
knowledge  about  the  Medicare  system: 

-  male  beneficiaries, 

•  white  beneficiaries, 

■  beneficiaries  with  higher  incomes, 

'    beneficiaries  with  more  education,  and 

■  beneficiaries  with  supplemental  insurance. 

•  beneficiaries  between  65  and  74  years  of  age 

-  beneficiaries  with  more  health  care  utilization. 

In  addition,  construct  validity  was  assessed  by  examining  the  correlation  between  the 
knowledge  indices.  A  high  correlafion  with  other  knowledge  scales  provides  addifional  evidence 
for  the  construct  validity  of  a  knowledge  scale. 

A  summary  of  the  psychometric  properties  of  these  knowledge  measures  is  presented  in 
Table  1.  (The  results  of  the  reliability  and  validity  analyses  are  presented  in  pages  10-21  of  the 
Knowledge  Index  Technical  Note.)  Of  the  four  different  knowledge  indices  analyzed,  only  the 
4-item  quiz  in  Round  1 8  and  the  Know  All  Need  To  Know  Index  in  Rounds  1 8  and  24  were 
found  to  be  acceptable  for  both  internal  consistency  reliability  and  construct  validity.  The  3-item 
quiz  in  Round  24,  while  similar  in  construction  to  the  4-item  quiz  in  Round  1 8,  was  found  not  to 
be  internally  consistent.  Increasing  the  number  of  items  on  the  3-item  and  4-item  quizzes  would 
likely  increase  their  reliability.  The  Medicare  understandability  question  was  found  not  to  be 
correlated  with  the  other  indices  tested,  so  it  may  be  measuring  a  somewhat  different  construct 
than  the  other  three  scales. 


7 


Table  1.  Psychometric  Properties  of  Knowledge  Scales 


Knowledge  Scale 

Internal 

Consistency 
(Reliability) 

Validity  -  Known 
Groups 
Comparisons 

Validity  - 
Correlations  with 
Other  Scales 

Medicare  understandability  question 

N/A 

+ 

Four-item  quiz 

* 

+ 

+ 

Three-item  quiz 

+ 

+ 

Know-all-need-to-know  index 

+ 

+ 

+ 

Note:  A  notation  of  "+"  indicates  that  the  scale  met  the  acceptable  level  for  this  criterion,  and  an  "-" 
denotes  the  failure  of  the  scale  to  reach  an  acceptable  level.  The  "*"  for  the  four-item  quiz 
indicates  that  this  scale  has  an  alpha  lower  than  0.70,  which  is  often  used  as  a  rule  of  thumb  for 
acceptable  internal  consistency.  However,  the  scale  has  an  alpha  greater  than  0.50,  which  may  be 
considered  promising  internal  consistency  for  a  scale  under  development. 


3.3     Recommended  MCBS  Baseline  Measures 

These  psychometric  tests  indicate  tliat  only  two  of  the  four  knowledge  indices  tested  are 
acceptable  and  recommended  for  use  as  baseline  measures  for  monitoring  the  effects  of  the 
NMEP  in  future  years: 

•  the  know-all-need-to-know  index  in  Roimd  1 8  of  the  1 996  MCBS,  and 

•  the  4-item  quiz  also  in  Round  1 8  of  the  1 996  MCBS. 

While  the  know-all-need-to-know  index  (to  be  called  the  Level  of  Perceived  Medicare 
Knowledge  Index  hereafter)  can  be  analyzed  over  MCBS  years  1996  and  1998  and  future  years, 
the  index  based  on  the  4-item  quiz  (hereafter  the  Level  of  Medicare  Knowledge  Index)  cannot  be 
used  over  time  as  the  number  of  useable  similar  questions  decreased  from  4  in  the  1996  MCBS 
round  18  to  3  in  the  1998  MCBS  round  24,  and  only  one  of  the  original  questions  carried  over 
from  1996  to  1998.  The  Level  of  Medicare  Knowledge  Index  in  1996  did  have  acceptable 
psychometric  properties  and  could  conceivably  be  used  for  comparisons  with  quizzes  containing 
larger  numbers  of  items  over  time  beginning  with  the  1999  MCBS  (if  these  items  or  similar  ones 
are  administered  in  ftiture  years). 

4.0    Descriptive  Analyses  Performed  on  Recommended  IVIeasures 

To  investigate  the  potential  for  the  NMEP  to  have  an  effect  on  beneficiary  knowledge, 
we  conducted  descriptive  analyses  of  the  measures  from  the  1996  and  1998  MCBS.  First,  we 
performed  frequency  distributions  for  the  Level  of  Medicare  Knowledge  and  Level  of  Perceived 
Medicare  Knowledge  items  for  years  1996  and  1998.  Preliminary  assessments  of  whether 
beneficiaries  have  increasing  knowledge  about  the  Medicare  program  over  time  can  be  made 
only  for  the  Level  of  Perceived  Medicare  Knowledge  measure  by  presenting  the  distributions  for 
each  year  side  by  side  because  the  questions  in  the  index  for  both  years  are  the  same.  The  Level 
of  Medicare  Knowledge  index  cannot  be  compared  between  the  two  years  in  this  fashion  as 
questions  in  the  measure  differ  between  1996  and  1998. 
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We  performed  bivariate  analyses  to  explore  how  knowledge  varies  across  beneficiary 
characteristics.  Chi-square  analyses  indicate  significant  differences  in  the  distribution  of 
knowledge  for  any  one  of  the  characteristics  analyzed.  Results  from  these  analyses  may  be  used 
to  make  suggestions  in  how  the  NMEP  is  implemented  in  order  to  increase  beneficiary 
knowledge  among  subgroups  who  may  score  substantially  lower  on  a  given  knowledge  measure. 
Selective  targeting  of  these  subgroups  may  be  a  cost-effective  way  of  improving  beneficiary 
knowledge  among  those  beneficiaries  whose  level  of  knowledge  needs  the  most  improvement. 

We  have  identified  several  conceptually  distinct  domains  that  we  tested  for  an  association 
with  the  need  for  information  about  aspects  of  the  Medicare  program.  These  domains  (shown  in 
Table  2)  consist  of  (1)  demographic  and  socio-economic  measures;  (2)  health  status  indicators; 
(3)  health  insurance  sources;  and  (4)  extent  of  health  services  utilization.  Various  studies  have 
tested  for  a  significant  association  between  variables  representing  these  domains  and  knowledge 
of  Medicare  beneficiaries  about  their  insurance.  Demographic  and  socio-economic  factors 
commonly  associate  1  with  higher  levels  of  knowledge  include  being  male  (Lambert,  1980), 
being  of  younger  age  (Lambert,  1980;  Caffereta,  1984;  McCall,  Rice,  and  Sangl,  1986),  being 
white  (Marquis,  1983;  McCall,  Rice,  and  Sangl,  1986),  being  of  higher  income  (Lambert,  1980; 
Marquis,  1983;  McCall,  Rice,  and  Sangl,  1986;  Rice,  McCall,  and  Boismier,  1990;  Hibbard,  et 
al.,  1998),  and  having  higher  education  (Caffereta,  1984;  McCall,  Rice,  and  Sangl,  1986; 
Hibbard,  et  al.,  1998;  Lambert,  1980;  Marquis,  1983;  McCall,  Rice,  and  Sangl,  1986). 

Although  health  status  has  not  been  previously  found  to  be  a  significant  predictor  of 
knowledge,  we  explored  this  relationship  using  the  MCBS  data.  We  used  three  different  health 
status  indicators — self-reported  health  status,  deficits  in  the  activities  of  daily  living,  and  whether 
a  proxy  answered  questions  for  the  beneficiary.  This  effort  will  help  in  imderstanding  what  to 
expect  when  we  include  health  status  variables  in  multivariate  modeling  in  the  Phase  2  report. 

Several  studies  have  explored  the  effect  of  types  of  insurance  coverage  on  knowledge. 
Marquis  (1983)  found  knowledge  to  be  significantly  associated  with  being  offered  a  choice  of 
health  plans  and  adversely  affected  by  plan  complexity.  Caffereta  (1984)  found  that  having 
supplemental  insurance  was  positively  related  to  knowledge.  She  also  found  that  the  use  of 
services  was  positively  associated  with  knowledge  in  a  model  subset  to  the  privately  insured, 
while  other  studies  have  not  found  such  an  effect.  In  addition,  three  utilization  variables  from 
the  MCBS— Part  A  use.  Part  B  use,  and  total  covered  charges — will  be  used  in  these  analyses  to 
determine  their  effect  on  knowledge. 

Based  on  findings  in  the  literature  previously  noted,  we  hypothesize  that  persons  who  are 
married,  male,  ages  65-75,  white,  of  higher  income,  fi-om  smaller  households,  and  with  higher 
education  will  have  higher  knowledge  scores.  We  hypothesize  that  persons  who  do  not  have  a 
proxy  respondent,  report  fewer  deficits  in  the  activities  of  daily  living,  and  present  better  self- 
reported  health  status  will  have  higher  knowledge  scores.  We  also  expect  that  persons  who  age 
into  Medicare,  have  both  Medicare  and  private  insurance,  and  have  managed  care  enrollment 
will  have  higher  knowledge  scores.  Finally,  we  hypothesize  that  persons  with  any  Part  A  or  Part 
B  use  and  higher  total  charges  will  have  higher  knowledge  scores. 
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Table  2.  MCBS  Variables  Used  in  the  Analysis  of  Characteristics 


Domain  and  Variable  Name 

Response  Levels  of  Variables 

Demographic  and  Socio-Economic  Characteristics 

Marital  Status 

1  =  Married 

2  =  Widowed 

3  =  Divorced/Separated 

4  =  Never  married 

Gender 

1  =  Male 

2  =  Female 

Age  Group 

1  =  Under  65  years  of  age 

2  =  65-75 

3  =  Over  75 

Race/Ethnicity 

1  =  White,  non-Hispanic 

2  =  Non-white,  non-Hispanic 

3  =  Hispanic 

Income  Level 

1  =  Under  $25,000 

2  =  $25,000  or  more 

Household  Size 

1  =  1  person 

2  =  2  persons 

3  =  3-4  persons 

4  =  5  or  more  persons 

Educational  Level 

1  =  8"  grade  or  less 

2  =  Some/completed  high  school 

3  =  Some/completed  college  or  more 

Health  Status 

Proxy  Status 

1  =  Not  a  proxy  interview 

2  =  Proxy 

Activities  of  Daily  Living 

1  =  No  limitations 

2  =  Limitations  in  one  or  two  activities 

3  =  Limitations  in  three  activities 

Perceived  Health  Status 

1  =  Excellent/Very  good 

2  =  Good 

3  =  Fair/Poor 

Health  Insurande  Coverage 

Medicare  Entitlement  Status 

1  =  Aged 

2  =  Disabled 

Type  of  Health  Insurance 

1  =  Medicare  only 

2  =  Medicare  and  private 

3  =  Medicare  and  Medicaid 

Managed  Care  Enrollment 

1  =  No  enrollment 

2  =  Enrolled  at  some  time 

Health  Services  Utilization 

Institutional  Utilization 

1  =  Any  utilization 

2  =  No  utilization 

Medicare  Part  B  Utilization 

1  =  No  claims 

2  =  Some  claims 

Total  Covered  Charges 

1  =$0 

2  =  $  1  -  $  499 

3  =  $  500  -  $  4,999 

4  =  $  5,000  or  more 

i 
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A  word  is  in  order  about  the  utilization  variables  as  they  have  some  limitations.  Because 
claims  data  do  not  typically  exist  for  beneficiaries  enrolled  in  Medicare  HMOs,  and  since  there 
are  no  self-report  data  in  the  MCBS  Access  to  Care  file  indicating  whether  there  has  been 
utilization  of  health  services,  we  have  no  choice  but  to  rely  upon  the  utilization  variables  that  are 
based  on  administrative  data.  We  believe  it  is  important  to  retain  managed  care  participants  in 
the  analysis,  but  concede  that  utilization  variables  based  on  administrative  data  are  somewhat 
crude  measures  and  only  incompletely  represent  utilization. 

5.0  Results 

5.1  Frequency  Distributions 

Weighted  frequency  distributions  for  the  1996  and  1998  MCBS  are  presented  in  Table  3 
below  with  associated  95%  confidence  intervals  for  the  Level  of  Medicare  Knowledge  Index. 


Table  3.  Distribution  of  the  Level  of  Medicare  Knowledge  Index  Scores  (Four  and  Three 
Item  Quizzes)  in  the  1996  and  1998  MCBS 


Level  of  Medicare  Knowledge 
(Four  and  Three  Item  Quizzes) 

1996  MCBS 

1998  MCBS 

High  (All  items  correct) 

18.7%  (17.0-19.7%) 

26.3%  (24.2-28.4%) 

Medium  (One  item  incorrect) 

39.2%  (37.9-40.5%) 

34.8%  (33.6-36.0%) 

Low  (Two  or  more  items  incorrect) 

42.1%  (40.3-43.8%) 

38.9%  (37.0-40.8%) 

Without  controlling  for  any  beneficiary  characteristics,  more  respondents  scored  in  the 
low  category  than  in  any  other  category  in  both  years.  Though  more  respondents  gave  correct 
answers  to  all  questions  in  the  3-item  quiz  in  the  1998  MCBS  than  in  the  4-item  quiz  in  the  1996 
MCBS,  one  can  not  compare  the  two  years  directly  as  the  number  and  content  of  the  questions 
differed  between  the  two  years  (i.e.  it  is  possible  that  the  1998  quiz  was  easier  and  that  is  why  a 
higher  percentage  got  all  items  correct). 

Frequency  distributions  of  the  Level  of  Perceived  Knowledge  of  Medicare  Index  are 
presented  in  Table  4.  The  results  presented  are  the  sum  of  the  Likert  values  given  to  five 
questions  about  beneficiary  perceived  knowledge  of  the  Medicare  program.  While  relatively  few 
persons  scored  high  in  either  year  (24.7  percent  in  1996  compared  to  22.0  percent  in  1998),  the 
persons  who  scored  at  least  high  or  medium  high  in  each  year  was  49.0  percent,  suggesting  that 
the  level  of  perceived  knowledge  has  remained  about  the  same  over  time. 


Table  4.  Distribution  of  the  Level  of  Perceived  Medicare  Knowledge  (Know-AU-Need-to- 
Know)  Index  Scores  in  the  1996  and  1998  MCBS 


Level  of  Perceived  Medicare 
Knowledge  (Five  Item  Knows  All 
Needs  to  Know  Index) 

1996  MCBS 

1998  MCBS 

High  (20  to  25) 

24.7  %  (22.5-26.8%) 

22.0%  (20.1-23.8%) 

Medium  High  (16  to  19) 

25.2%  (23.7-26.7%) 

27.9%  (26.8-29.1%) 

Medium  Low  (12  to  15) 

23.8  %  (22.8-24.9%) 

25.9%  (24.7-27.2%) 

Low  (5  to  11) 

26.3  %  (24.4-28.1%) 

24.2%  (22.4-25.9%) 
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A  test  of  significance  between  the  distribution  of  scores  in  1996  and  1998  for  this  index  using  a 
chi-square  showed  a  significant  difference  between  the  years.  It  appears  that  regression  to  the 
mean  may  be  occurring  as  the  middle  two  categories  combined  increased  by  about  5  percent  in 
1998  and  the  extreme  categories  each  decreased  by  about  2  percent  from  their  values  in  1996. 

5.2     Cross  Tabulations  of  Knowledge  Measures  with  MCBS  Beneficiary 
Characteristics 

The  Medicare  Knowledge  Index  and  the  Perceived  Medicare  Knowledge  Index  were 
each  compared  with  the  1 6  beneficiary  attributes  described  in  Table  2  using  Chi-square  analyses 
to  test  for  statistically  significant  differences  at  a  conventional  level  of  significance  (p<=0.05). 
Only  significant  differences  are  discussed  in  the  text. 

5.2.1  Tests  of  Significance  between  Level  of  Medicare  Knowledge  Index  and 
MCBS  Respondent  Characteristics 

For  the  Medicare  Knowledge  Index,  bivariate  associations  with  beneficiary 
characteristics  were  analyzed  using  three  response  levels  that  were  created  for  this  knowledge 
measure.  In  order  to  have  a  common  metric  for  each  of  the  two  years  analyzed,  we  created  a 
categorical  response  variable  with  three  levels: 

■  l=high  (all  questions  in  the  knowledge  index  were  answered  correctly), 
•    2=medium  (only  one  incorrect  response  was  given),  and 

■  3=low  (two  or  more  incorrect  responses  were  given). 

While  this  metric  is  common  to  both  years,  one  cannot  compare  directly  the  results  across 
the  years  as  the  1996  version  of  the  index  contains  four  items  and  the  1998  version  contains  only 
three  items.  Further,  only  one  of  the  questions  was  common  between  the  two  years. 

Table  5  shows  that  there  are  significant  differences  between  the  levels  of  Medicare 
knowledge  for  all  but  one  of  the  beneficiary  characteristics  examined  for  both  1996  and  1998  for 
the  Level  of  Medicare  Knowledge  Index.  Gender  was  the  only  characteristic  for  which  there  was 
not  a  significant  difference  among  beneficiaries  for  this  measure,  and  this  was  only  in  1996.  It  is 
not  surprising  that  so  many  characteristics  were  significant  from  the  Chi-square  analyses  given 
the  large  sample  size.  Small  percentage  differences  for  frequency  distributions  among  such  a 
large  sample  can  lead  to  findings  of  statistical  significance  that  might  otherwise  not  be  found  in 
smaller  samples. 
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Table  5.  Summary  of  Chi-square  Significance  Tests  between  Medicare  Beneficiary 

Characteristics  and  Level  of  Medicare  Knowledge  in  the  1996  and  1998  MCBS 


Year  of  MCBS 

1996 

1998 

Chi-square 

Degrees  of 

Chi-square 

Degrees  of 

Variable 

value 

freedom 

p-value 

value 

freedom 

p-value 

Demographic  and  Socio-Economic  Characteristics 

Marital  Status 

179.20 

6 

<.001 

210.95 

6 

<  .001 

Gender 

4.42 

2 

n.s. 

7.28 

2 

.027 

Age  Group 

141.74 

4 

<.001 

214.57 

4 

<.001 

Race/Ethnicity 

214.47 

4 

<.001 

172.49 

4 

<.001 

Income  Level 

127.89 

2 

<.001 

247.55 

2 

<.001 

Household  Size 

213.52 

6 

<.001 

216.67 

6 

<.001 

Education  Level 

232.02 

4 

<.001 

339.39 

4 

<.001 

Health  Status 

Proxy  Status 

7.83 

2 

.020 

7.13 

2 

.029 

Activities  of  Daily 

31.73 

4 

<.001 

37.06 

4 

<.001 

Living 

Perceived  Health  Status 

106.62 

4 

<.001 

66.42 

4 

<  .001 

Health  Insurance  Coverage 

Medicare  Entitlement 

130.63 

2 

<  .001 

137.07 

2 

<  .001 

Type  of  Insurance 

505.81 

4 

<  .001 

330.66 

4 

<.001 

Managed  Care 

39.48 

2 

<  .001 

17.55 

2 

<.001 

Health  Services  Utilization 

Institutional  Utilization 

94.22 

2 

<.00I 

24.84 

2 

<  .001 

Part  B  Utilization 

278.21 

2 

<.001 

74.16 

2 

<.001 

Total  Covered  Charges 

226.61 

6 

<  .001 

93.32 

6 

<.001 

In  Tables  6  through  9  we  present  detail  about  the  magnitude  of  these  results  for  the 
Medicare  Knowledge  Index  grouped  by  each  of  the  four  domains:  demographic  and  socio- 
economic characteristics,  health  status,  health  insurance  coverage,  and  health  services  utilization. 
Given  that  almost  all  of  the  Chi-Square  tests  were  significant  as  a  result  of  the  large  sample  size, 
absolute  differences  in  magnitude  may  be  more  important  for  policy  implications.  Following 
presentation  of  all  distributional  analyses  for  the  Medicare  Knowledge  Index,  the  distributional 
analyses  for  the  Perceived  Medicare  Knowledge  Index  are  presented  in  Tables  10  through  14. 

Demographic  and  Socio-Economic  Characteristics 

All  seven  of  the  Medicare  beneficiary  demographic  and  socio-economic  characteristics 
analyzed  were  significant  in  both  1996  and  1998,  except  for  gender  that  was  only  significant  in 
1996.  The  distributions  of  levels  of  Medicare  knowledge  for  each  Medicare  beneficiary 
demographic  and  socio-economic  characteristic  are  presented  in  Table  6. 
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Table  6.  Percentage  Distributions  of  Beneficiary  Demographic  and  Socio-economic 
Characteristics  and  Levels  of  Medicare  Knowledge  Index  Scores  in  the 
1996  and  1998  MCBS 


Year  of  MCBS 

1996 

1998 

Variable 

High 

Medium 

Low 

High 

Medium 

Low 

Marital  Status 

Married 

22.18 

41.68 

36.14 

30.72 

35.72 

33.56 

Widowed 

15.83 

37.76 

46.41 

21.81 

34.82 

43.38 

Divorced/ 
Separated 

11.68 

33.54 

54.78 

18.83 

32.45 

48.72 

Never  Married 

13.19 

32.48 

54.33 

19.33 

30.40 

50.27 

Gender 

Male 

19.54 

38.15 

42.32 

27.32 

33.43 

39.25 

Female 

18.11 

40.01 

41.88 

25.45 

35.87 

38.69 

Age  Group 

Under  65  years 

11.37 

30.74 

57.89 

17.09 

29.59 

53.31 

65  to  75  years 

20.28 

41.39 

38.33 

29.93 

35.99 

34.09 

Over  75  years 

18.81 

38.65 

42.55 

24.44 

34.94 

40.62 

Race/Ethnicity 

White  Non-Hispanic 

20.86 

41.05 

38.08 

29.18 

36.25 

34.56 

Non- White 
Non-Hispanic 

6.75 

31.41 

61.85 

13.77 

27.53 

58.70 

Hispanic 

11.32 

27.91 

60.77 

12.23 

29.01 

58.76 

Income  Level 

Less  than  $25,000 

15.93 

37.77 

46.30 

21.61 

33.99 

44.40 

$25,000  or  more 

25.08 

42.41 

32.51 

35.89 

36.43 

27.68 

Household  size 

1  person 

15.76 

37.28 

46.96 

22.67 

34.73 

42.59 

2  persons 

22.15 

41.83 

36.02 

30.29 

36.15 

33.56 

3-4  persons 

14.19 

35.35 

50.47 

21.58 

30.85 

47.58 

5  or  more  persons 

10.33 

30.12 

59.55 

13.69 

29.29 

57.02 

Education  Level 

College 

24.88 

41.74 

33.37 

35.70 

35.63 

28.67 

High  School 

18.76 

40.72 

40.51 

25.07 

36.03 

38.90 

8""  grade  or  less 

10.89 

32.59 

56.52 

15.08 

30.47 

54.45 

In  both  years,  married  persons  scored  higher  on  Medicare  knowledge  than  other  persons 
and  persons  who  were  divorced/separated  scored  the  lowest.  Gender  was  only  significant  in 
1998,  where  there  is  a  larger  variation  in  response  for  males  than  females,  though  there  is  little 
difference  between  the  groups.  In  both  years,  persons  aged  65-75  scored  higher  than  other 
persons,  and  persons  under  age  65  scored  the  lowest. 

In  both  years,  white  non-Hispanic  persons  scored  higher  than  other  persons,  and  both 
non-white  non-Hispanic  and  Hispanic  persons  scored  very  close  together  for  the  lowest.  In  both 
years,  persons  with  $25,000  or  more  in  income  scored  higher  than  persons  with  less  than  $25,000 
in  income.  In  both  years,  persons  in  households  with  two  persons  scored  higher  than  persons  in 
other  categories,  and  persons  with  five  or  more  persons  in  the  household  scored  the  lowest.  In 
both  years,  persons  with  college  experience  scored  higher  than  other  persons,  and  persons 
finishing  8'*'  grade  or  less  scored  the  lowest. 
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Health  Status  Characteristics 

All  three  of  the  Medicare  beneficiary  health  status  characteristics  analyzed  were 
significant  in  both  1996  and  1998.  The  results  for  each  Medicare  beneficiary  health  status 
characteristic  are  reported  below  in  Table  7. 

Table  7.  Percentage  Distributions  of  Beneficiary  Health  Status  Characteristics 
and  Levels  of  Medicare  Knowledge  Index  Scores  in  the  1996  and  1998 
MCBS 


Yearof  MCBS 

1996 

1998 

Variable 

High 

Medium 

Low 

High 

Medium 

Low 

Proxy  Status 

Sample  Person 

18.98 

39.38 

41,64 

26.34 

35.11 

38.55 

Proxy 

16.22 

37.20 

46.58 

25.56 

31.32 

43.12 

Activities  of  Daily  Livinj 

No  limitations 

20.07 

39.21 

40.72 

27.51 

35.04 

37.45 

Limitations  in  one  or 
two  activities 

15.58 

39.69 

44.73 

24.72 

33.78 

41.51 

Limitations  in  three  or 
more  activities 

14.09 

39.28 

46.62 

18.97 

35.87 

45.16 

Perceived  Health  Status 

Very  Good/Excellent 

20.78 

40.43 

38.79 

29.26 

35.72 

35.02 

Good 

19.39 

39.56 

41.05 

26.41 

35.30 

38.29 

Fair/Poor 

14.39 

36.59 

49.02 

21.50 

32.75 

45.75 

In  both  years,  sample  persons  scored  higher  on  Medicare  knowledge  than  proxy  persons. 
In  both  years,  persons  with  no  limitations  scored  higher  than  other  persons,  and  persons  with 
limitations  in  three  or  more  activities  scored  the  lowest.  In  both  years,  persons  reporting  very 
good  or  excellent  health  scored  higher  than  other  persons,  and  persons  reporting  fair  or  poor 
health  scored  the  lowest.  These  consistent  findings  suggest  that  persons  who  are  in  better  health 
have  greater  knowledge  of  the  Medicare  program. 

Health  Insurance  Coverage  Characteristics 

All  three  of  the  Medicare  beneficiary  health  insurance  coverage  characteristics  analyzed 
were  significant  in  both  1996  and  1998  as  shown  in  Table  8. 
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Table  8.  Percentage  Distributions  of  Beneficiary  Health  Insurance  Coverage 
Characteristics  and  Levels  of  Medicare  Knowledge  Index  Scores  in 
the  1996  and  1998  MCBS 


YearofMCBS 

1996 

1998 

Variable 

High 

Medium 

Low 

High 

Medium 

Low 

Medicare  Entitlement 

Aged 

19.69 

40.29 

40.02 

27.65 

35.57 

36.78 

Disabled 

11.42 

30.79 

57.79 

16.98 

29.51 

53.52 

Type  of  Insurance 

Medicare  Only 

12.37 

31.63 

56.00 

20.60 

32.84 

46.56 

Medicare  and  Private 

22.73 

42.43 

34.84 

31.10 

36.49 

32.42 

Medicare  and  Public 

8.82 

33.70 

57.48 

15.07 

30.76 

54.17 

Managed  Care 

Not  Enrolled 

19.50 

39.00 

41.49 

26.17 

34.19 

39.63 

Enrolled 

11.16 

41.07 

47.77 

26.95 

38.63 

34.42 

In  both  years,  beneficiaries  receiving  Medicare  because  they  are  aged  scored  higher  on 
Medicare  knowledge  than  beneficiaries  whose  program  entitlement  resulted  from  being  disabled. 
In  both  years,  persons  with  Medicare  and  private  insurance  scored  higher  than  other  persons,  and 
persons  with  Medicare  and  public  insurance  scored  the  lowest.  In  1996,  persons  not  enrolled  in 
managed  care  scored  higher  on  Medicare  knowledge  than  persons  enrolled  in  managed  care. 
However,  in  1998,  persons  enrolled  in  managed  care  scored  slightly  higher  than  persons  not 
enrolled  in  managed  care,  but  the  scores  were  very  close. 

Health  Services  Utilization  Characteristics 

All  three  of  the  health  services  utilization  characteristics  of  Medicare  beneficiaries 
analyzed  were  significant  in  both  1996  and  1998,  and  their  distributions  are  shown  in  Table  9. 

Table  9.  Percentage  Distributions  of  Beneficiary  Health  Services  Utilization 
Characteristics  and  Levels  of  Medicare  Knowledge  Index  Scores  in 
the  1996  and  1998  MCBS 


Year  of  MCBS 

1996 

1998 

Variable 

High 

Medium 

Low 

High 

Medium 

Low 

Institutional  Utilization 

Yes 

21.07 

41.33 

37.59 

27.72 

35.70 

36.58 

No 

15.42 

36.16 

48.41 

24.34 

33.56 

42.10 

Medicare  Part  B  Utilization 

No  Claims 

8.76 

31.31 

59.93 

21.09 

33.32 

45.59 

Some  Claims 

20.60 

40.67 

38.74 

27.77 

35.21 

37.01 

Total  Covered  Charges 

$0 

9.17 

31.43 

59.39 

21.11 

33.28 

45.61 

$1-499 

20.25 

38.93 

40.83 

25.31 

33.05 

41.63 

$500-4,999 

20.60 

41.72 

37.68 

28.50 

36.65 

34.85 

$5,000  or  More 

20.78 

40.85 

38.38 

28.37 

34.74 

36.89 
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In  both  years,  persons  with  institutional  utilization  (Part  H)  scored  higher  on  Medicare 
knowledge  than  persons  without  institutional  utilization.  In  both  years,  persons  with  some  Part  B 
claims  scored  higher  than  persons  without  Part  B  claims.  In  both  years,  persons  with  charges  in 
any  category  scored  higher  than  persons  without  charges.  The  consistent  findings  suggest  that 
persons  who  have  experience  using  Medicare  are  better  informed  about  it. 

5.2.2  Tests  of  Significance  between  Levels  of  Perceived  Medicare  Knowledge 
and  MCBS  Respondent  Characteristics 

Table  10  shows  the  tests  of  significance  for  the  Perceived  Medicare  Knowledge  Index. 
Most  of  the  beneficiary  characteristics  were  significant  in  both  years,  with  the  exception  of 
institutional  utilization  that  was  not  significant  in  either  year,  gender  that  was  not  significant  in 
1998,  and  proxy  status  that  was  not  significant  in  1998. 

Table  10.  Summary  f^f  Chi-square  Significance  Tests  between  Medicare 

Beneficiary  Characteristics  and  Level  of  Perceived  Medicare  Knowledge 
in  the  1996  and  1998  MCBS 


Year  of  MCBS 

1996 

1998 

Variable 

Chi-square 
value 

Degrees  of 
freedom 

p-value 

Chi-square 
value 

Degrees 

of 
freedom 

p-value 

Demographic  and  Socio-Economic  Characteristics 

Marital  Status 

160.79 

9 

<.001 

265.75 

9 

<.001 

Gender 

24.30 

3 

<  .001 

4.71 

3 

n.s. 

Age  Group 

140.95 

6 

<.001 

161.16 

6 

<.001 

Race/Ethnicity 

109.22 

6 

<.001 

131.58 

6 

<.001 

Income  Level 

301.67 

3 

<.001 

357.35 

3 

<.001 

Household  Size 

145.73 

9 

<.001 

249.73 

9 

<.001 

Education  Level 

283.21 

6 

<  .001 

315.16 

6 

<  .001 

Health  Status 

Proxy  Status 

9.41 

3 

.025 

5.94 

3 

n.s. 

Activities  of  Daily 
Living 

88.74 

6 

<.001 

89.74 

6 

<.001 

Perceived  Health 
Status 

135.93 

6 

<.001 

193.66 

6 

<.001 

Health  Insurance  Coverage 

Medicare  Entitlement 

121.57 

3 

<  .001 

132.33 

3 

<.00I 

Type  of  Insurance 

342.50 

6 

<.001 

346.96 

6 

<.001 

Managed  Care 

55.24 

3 

<  .001 

46.48 

3 

<  .001 

Health  Services  Utilization 

Institutional  Utilization 

5.40 

3 

n.s. 

2.16 

3 

n.s. 

Part  B  Utilization 

31.75 

3 

<.001 

23.03 

3 

<.001 

Total  Covered  Charges 

54.11 

9 

<.001 

32.01 

9 

<.001 

Tables  1 1  through  14  show  the  distribution  of  results  for  each  of  the  four  domains  of 
beneficiary  characteristics  in  the  analysis:  demographic  and  socio-economic  characteristics, 
health  status,  health  insurance  coverage,  and  health  services  utilization. 
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Demographic  and  Socio-Economic  Characteristics 

All  seven  of  the  beneficiary  demographic  and  socio-economic  characteristics  analyzed 
were  significant  in  both  the  1996  and  1998  MCBS,  except  for  gender,  that  was  only  significant  in 
1996.  The  results  for  each  of  these  characteristic  are  described  below  in  Table  11. 


Table  11.  Percentage  Distributions  of  Beneficiary  Demographic  and  Socio-economic  Characteristics 
and  Levels  of  Perceived  Medicare  Knowledge  Index  Scores  in  the  1996  and  1998  MCBS 


Year  of  MCBS 

1996 

1998 

Variable 

High 

Medium 
High 

Medium 

Low 

Low 

High 

Medium 
High 

Medium 

Low 

Low 

Marital  Status 

Married 

27.17 

27.91 

23.74 

21.18 

24.83 

30.82 

25.10 

19.24 

Widowed 

23.00 

22.42 

23.61 

30.97 

19.73 

25.81 

26.40 

28.06 

Divorced/ 
Separated 

17.49 

22.42 

24.78 

35.31 

15.66 

22.85 

27.03 

34.47 

Never  Married 

21.18 

19.50 

24.68 

34.64 

16.77 

20.14 

29.61 

33.48 

Gender 

Male 

26.51 

25.55 

21.93 

26.02 

22.96 

27.71 

25.37 

23.96 

Female 

23.24 

24.95 

25.36 

26.45 

21.18 

28.09 

26.42 

24.31 

Age  Group 

Under  65  years 

17.25 

18.77 

25.26 

38.72 

14.16 

20.73 

29.69 

35.42 

65  to  75  years 

25.76 

27.30 

23.75 

23.19 

24.19 

29.90 

25.16 

20.74 

Over  75  years 

25.44 

24.13 

23.56 

26.87 

21.62 

27.69 

25.74 

24.95 

Race/Ethnicity 

White  Non-Hispanic 

26.19 

27.12 

23.60 

23.08 

24.41 

30.09 

25.57 

19.93 

Non- White 
Non-Hispanic 

18.28 

16.39 

25.65 

39.69 

12.00 

19.00 

26.98 

42.02 

Hispanic 

15.55 

15.00 

24.05 

45.40 

9.17 

17.04 

28.85 

44.94 

Income  Level 

Less  than  $25,000 

21.16 

23.00 

24.61 

31.23 

18.10 

25.11 

27.32 

29.48 

$25,000  or  more 

32.57 

30.17 

22.15 

15.10 

29.93 

33.71 

23.14 

13.22 

Household  size 

1  person 

22.91 

22.86 

23.73 

30.49 

19.63 

26.17 

26.17 

28.04 

2  persons 

26.48 

27.78 

23.50 

22.24 

24.81 

30.37 

25.25 

19.57 

3-4  persons 

22.71 

21.20 

25.52 

30.56 

17.53 

24.27 

28.08 

30.11 

5  or  more  persons 

19.89 

22.22 

23.86 

34.04 

15.70 

19.18 

26.60 

38.52 

Education  Level 

College 

33.39 

29.09 

21.75 

15.77 

28.50 

33.61 

23.20 

14.69 

High  School 

23.46 

26.38 

24.76 

25.40 

21.93 

28.08 

27.03 

22.96 

8""  grade  or  less 

16.25 

17.68 

24.54 

41.53 

12.23 

18.98 

27.45 

4L33 
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In  both  years,  married  persons  scored  higher  on  perceived  Medicare  knowledge  than 
other  persons,  and  persons  who  were  divorced/separated  or  never  married  scored  the  lowest.  The 
highest  frequency  reported  for  the  high  response  category  was  27.2  percent  which  was  reported 
by  married  persons  in  1996.  Gender  was  only  significant  in  1996,  where  males  scored  only 
slightly  higher  than  females,  though  there  is  little  difference  between  the  groups.  The  highest 
frequency  reported  by  gender  for  the  high  response  category  was  26.5  percent  which  was 
achieved  by  males  in  1996.  In  both  years,  persons  aged  65  to  75  scored  higher  than  other 
persons,  and  persons  under  age  65  scored  the  lowest.  The  highest  frequency  reported  in  the  high 
perceived  Medicare  knowledge  response  category  was  25.8  percent  which  was  scored  by  persons 
aged  65  to  75  in  1996. 

In  both  years,  white  non-Hispanic  persons  scored  higher  than  other  persons,  and  Hispanic 
persons  scored  the  lowest.  The  highest  frequency  reported  in  the  high  perceived  Medicare 
knowledge  category  was  26.2  percent  and  it  was  scored  by  white  non-Hispanic  persons  in  1996. 
In  both  years,  persons  with  $25,000  or  more  in  income  scored  higher  on  perceived  Medicare 
knowledge  than  persons  with  less  than  $25,000  in  income.  The  highest  frequency  reported  for 
the  high  perceived  knowledge  category  was  32.6  percent  and  it  was  scored  by  persons  with 
$25,000  or  more  in  income  in  1996.  In  both  years,  persons  in  households  with  two  persons 
scored  higher  than  persons  in  other  categories,  and  persons  with  five  or  more  in  their  household 
scored  the  lowest.  The  highest  frequency  reported  in  the  high  perceived  knowledge  category 
was  26.5  percent  and  it  was  scored  by  persons  in  households  with  two  persons  in  1996.  In  both 
years,  persons  with  college  experience  scored  higher  than  other  persons,  and  persons  finishing  8 
grade  or  less  scored  the  lowest.  The  highest  frequency  reported  in  the  high  perceived  knowledge 
category  was  33.4  percent  which  was  scored  by  persons  with  college  experience  in  1996. 

The  direction  of  association  between  perceived  Medicare  knowledge  and  beneficiary 
demographic  and  socio-economic  characteristics  as  consistent  across  the  two  years.  It  is  also 
worth  noting  that  the  proportions  in  the  high  category  of  perceived  Medicare  knowledge  for  each 
characteristic  were  higher  in  1996  than  in  1998,  the  only  exception  being  for  married  persons. 
The  results  suggest  that  with  respect  to  perceived  knowledge  of  Medicare,  the  proportions  of 
persons  scoring  at  the  highest  level  have  fallen  over  the  two  years. 

Health  Status  Characteristics 

All  three  of  the  beneficiary  health  status  characteristics  analyzed  were  significant  in  both 
the  1996  and  1998  MCBS  except  for  proxy  status,  that  was  only  significant  in  1996.  The  results 
for  each  characteristic  are  described  below  in  Table  12. 
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Table  12.  Percentage  Distributions  of  Beneficiary  Health  Status  Characteristics  and  Levels  of 
Perceived  Medicare  Knowledge  Index  Scores  in  the  1996  and  1998  MCBS 


YearofMCBS 

1996 

1998 

Variable 

High 

Medium 
High 

Medium 
Low 

Low 

High 

Medium 
High 

Medium 
Low 

Low 

Proxy  Status 

Sample  Person 

24.80 

25.47 

23.85 

25.88 

22.16 

28.02 

25.93 

23.89 

Proxy 

23.38 

22.45 

23.90 

30.27 

19.93 

26.88 

26.19 

27.01 

Activities  of  Daily  Livinj 

No  limitations 

26.53 

26.26 

23.17 

24.03 

24.16 

28.90 

24.90 

22.04 

Limitations  in  one  or 
two  activities 

20.10 

23.70 

26.71 

29.48 

17.25 

26.18 

27.58 

28.99 

Limitations  in  three  or 
more  activities 

18.86 

19.77 

24.69 

36.69 

14.69 

23.27 

31.14 

30.90 

■  Perceived  Health  Status 

Very  Good/Excellent 

28.32 

27.23 

23.69 

20.77 

26.35 

31.30 

23.49 

18.86 

Good 

23.74 

26.02 

24.09 

26.16 

21.54 

27.64 

27.16 

23.66 

Fair/Poor 

19.30 

20.85 

23.79 

36.06 

15.65 

23.02 

28.36 

32.97 

Proxy  status  was  only  significant  in  the  1996  MCBS  where  sample  persons  scored  higher 
on  perceived  knowledge  than  proxy  persons.  The  highest  proportion  reported  for  the  high 
perceived  knowledge  category  was  24.8  percent  that  was  reported  by  sample  persons  in  1996.  In 
both  years,  persons  with  no  limitations  scored  higher  than  other  persons,  and  persons  with 
limitations  in  three  or  more  activities  scored  the  lowest.  The  highest  frequency  reported  for  the 
high  perceived  knowledge  category  was  26.5  percent  that  was  reported  by  persons  with  no 
limitations  in  1996.  In  both  years,  persons  reporting  very  good  or  excellent  health  had  a  higher 
proportion  in  the  high  perceived  knowledge  category  than  other  persons,  and  persons  reporting 
fair  or  poor  health  had  the  largest  proportion  in  the  low  category.  The  highest  proportion 
reported  for  the  high  perceived  knowledge  category  was  28.3  percent  that  was  achieved  by 
Medicare  beneficiaries  in  1996  reporting  very  good  or  excellent  health.  Quite  consistently, 
larger  proportions  of  persons  in  better  health  scored  higher  on  perceived  Medicare  knowledge  in 
the  1996  MCBS  than  in  the  1998  MCBS. 

Health  Insurance  Coverage  Characteristics 

All  three  of  the  health  insurance  coverage  characteristics  analyzed  were  significant  in 
both  1996  and  1998.  The  results  for  each  characteristic  are  described  below  in  Table  13. 
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Table  13.  Percentage  Distributions  of  Beneficiary  Health  Insurance  Coverage  Characteristics  and 
Levels  of  Perceived  Medicare  Knowledge  Index  Scores  in  the  1996  and  1998  MCBS 


Yearof  MCBS 

1996 

1998 

Medium 

Medium 

Medium 

Medium 

Variable 

High 

High 

Low 

Low 

High 

High 

Low 

Low 

IMedicare  Entitlement 

Aged 

25.63 

26.03 

23.67 

24.67 

23.12 

28.98 

25.39 

22.50 

Disabled 

17.30 

18.84 

25.27 

38.60 

14.15 

20.72 

29.74 

35.38 

Tvoe  of  Insurance 

Medicare  Only 

21.48 

19.66 

22.54 

36.32 

18.66 

25.21 

25.44 

30.69  . 

Medicare  and  Private 

27.50 

28.70 

24.20 

19.60 

25.43 

31.31 

26.03 

17.24 

Medicare  and  Public 

16.19 

16.43 

23.80 

43.57 

12.86 

18.26 

26.36 

42.52 

Managed  Care 

Not  Enrolled 

23.71 

25.18 

23.97 

27.13 

21.09 

27.21 

26.53 

25.17 

Enrolled 

34.22 

25.47 

22.66 

17.64 

27.00 

32.49 

22.23 

17.61 

In  both  years,  persons  who  were  receiving  Medicare  because  they  were  aged  (65  years  of 
age  or  older)  scored  significantly  higher  than  those  under  65  whose  entitlement  was  due  to  being 
disabled.  The  highest  proportion  reported  in  the  high  perceived  Medicare  knowledge  category 
was  25.6  percent  that  was  reported  by  aged  Medicare  beneficiaries  in  1996.  In  both  years, 
persons  with  Medicare  and  private  insurance  scored  significantly  higher  than  other  persons,  and 
persons  with  Medicare  and  public  insurance  scored  the  lowest.  The  highest  proportion  reported 
in  the  high  perceived  Medicare  knowledge  category  was  27.5  percent  that  was  achieved  by 
persons  with  Medicare  and  private  insurance  in  1996.  In  both  years,  persons  enrolled  in 
managed  care  scored  higher  on  perceived  Medicare  knowledge  than  persons  not  enrolled  in 
managed  care.  The  highest  proportion  reported  in  the  high  perceived  Medicare  knowledge 
category  was  34.2  percent  that  was  attained  by  persons  enrolled  in  managed  care  in  1996. 
Consistently,  respondents  to  the  insurance  characteristic  items  in  the  1996  MCBS  had  higher 
proportions  reporting  a  high  level  of  perceived  knowledge  than  those  responding  in  the  1998 
MCBS. 

Health  Services  Utilization  Characteristics 

Only  two  of  the  three  health  services  utilization  characteristics  analyzed  were 
significantly  associated  with  perceived  Medicare  knowledge  in  both  1996  and  1998 — Part  B 
Utilization  and  Total  Covered  Charges.  Institutional  Charges  did  not  differ  significantly  by  level 
of  perceived  Medicare  knowledge  in  either  year.  The  results  for  each  characteristic  are  described 
below  in  Table  14. 
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Table  14.  Percentage  Distributions  of  Beneficiary  Health  Services  Utilization  Characteristics  and 
Levels  of  Perceived  Medicare  Knowledge  Index  Scores  in  the  1996  and  1998  MCBS 


Yearof  MCBS 

1996 

1998 

Variable 

High 

Medium 
High 

Medium 
Low 

Low 

High 

Medium 
High 

Medium 
Low 

Low 

Institutional  Utilization 

Yes 

25.63 

25.40 

23.61 

25.35 

21.90 

27.93 

26.45 

23.72 

No 

23.31 

24.93 

24.19 

27.56 

22.07 

27.91 

25.28 

24.74 

Medicare  Part  B  Utilization 

No  Claims 

21.11 

22.50 

22.87 

33.52 

20.92 

26.06 

24.60 

28.42 

Some  Claims 

25.33 

25.71 

24.03 

24.92 

22.27 

28.46 

26.34 

22.93 

Total  Covered  Charges 

$0 

22.27 

22.88 

22.39 

32.46 

21.17 

26.13 

24.69 

28.02 

$1-499 

24.56 

24.21 

24.76 

26.47 

20.97 

27.26 

26.62 

25.15 

$500-4,999 

25.72 

26.71 

24.58 

22.99 

22.89 

29.39 

26.19 

21.52 

$5,000  or  More 

24.77 

25.50 

22.66 

27.07 

22.12 

27.79 

26.14 

23.94 

In  both  years,  Medicare  beneficiaries  with  some  Part  B  claims  scored  higher  on  perceived 
Medicare  knowledge  than  those  without  any  Part  B  claims.  The  highest  fi-equency  reported  for 
the  high  category  of  perceived  Medicare  knowledge  was  25.3  percent  which  was  attained  by 
persons  with  some  Part  B  claims  in  1996.  In  both  years,  persons  with  total  charges  between 
$500  and  $4,999  scored  higher  than  persons  in  other  categories.  The  highest  frequency  reported 
for  the  high  perceived  Medicare  knowledge  category  was  approximately  25.7  percent  which  was 
reported  by  persons  with  total  charges  between  $500  and  $4,999  in  1996.  As  with  most  of  the 
other  characteristics,  higher  proportions  of  beneficiaries  in  the  1996  MCBS  were  in  the  higher 
perceived  Medicare  knowledge  category  than  in  the  1998  MCBS. 

5.3     Summary  of  Analyses 

There  were  significant  differences  in  almost  all  distributions  tested  when  comparing 
knowledge  measures  with  beneficiary  characteristics.  The  significance  of  these  differences  may 
be  the  result  of  the  large  MCBS  sample  sizes  for  which  small  differences  in  a  percentage 
distribution  will  yield  a  statistically  significant  test  result.  Persons  who  were  hypothesized  to  be 
more  knowledgeable  generally  had  the  most  favorable  knowledge  outcomes.  Persons  who  were: 
married,  male,  65  to  75  years  of  age,  white  non-Hispanic,  with  more  than  $25,000  in  income,  in 
households  with  two  persons,  and  college  educated  had  greater  knowledge  about  Medicare.  In 
addition,  persons  who  did  not  have  a  proxy,  had  no  limitations  in  the  activities  of  daily  living, 
and  reported  very  good  or  excellent  health  status  were  more  knowledgeable  about  Medicare. 
Finally,  persons  without  disabilities,  who  had  Medicare  and  private  insurance,  and  had  utilization 
of  some  kind  also  had  more  Medicare  knowledge. 

In  general,  persons  who  were:  divorced,  separated,  or  otherwise  living  alone;  under  age 
65;  of  minority  group  status;  with  lower  income;  in  larger  households;  with  less  than  an  8  grade 
education;  with  several  limitations  in  activities  of  daily  living;  in  fair  or  poor  health;  disabled, 
with  both  Medicare  and  Medicaid;  and  not  using  health  services  had  less  favorable  Medicare 
knowledge  index  scores.  These  persons  may  be  the  most  appropriate  persons  toward  which  to 
target  knowledge  improvement  efforts.  Changes  to  the  NMEP  as  currently  implemented  would 
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more  likely  be  beneficial  with  respect  to  characteristics  where  there  was  a  large  variance  in  the 
distribution  of  responses,  and  then  targeted  to  those  individuals  who  scored  much  lower  than 
other  persons.  An  example  of  a  characteristic  with  a  large  variance  is  Type  of  Health  Insurance, 
where  persons  with  both  Medicare  and  Medicaid  scored  substantially  lower  on  Medicare 
knowledge  measures  than  persons  in  other  insurance  categories. 

6.0  Related  Knowledge  Indices  and  Questions  Available  for 
Monitoring  NMEP 

6.1  Cross-Project  Knowledge  Indices 

RTI  is  conducting  two  other  studies  for  HCFA  in  which  it  has  created  and  evaluated 
indices  to  measure  knowledge  in  the  Medicare  population.  These  studies  include  the  Evaluation 
of  Medicare  CAHPS  and  the  Medicare  &  You  1999  Handbook  in  Kansas  City  and  the  Expanded 
(National)  Evaluation  of  the  Medicare  &  You  2000  Handbook.  For  each  of  these  evaluations, 
RTI  created  a  knowledge  index  to  assess  how  well  beneficiaries  understood  the  Medicare 
program  and  related  health  insurance  options.  The  Kansas  City  knowledge  index  contained  15 
items  and  the  National  Evaluation  index  contained  23  items.  The  indices  were  tailored  to  reflect 
the  information  in  the  handbook  during  the  respective  year  as  well  as  the  key  messages  HCFA 
was  promoting  at  the  time. 

Both  indices  proved  valid  and  reliable  using  standard  psychometric  tests  and  have  been 
used  as  dependent  variables  in  multivariate  analysis  to  determine  the  factors  statistically 
associated  with  higher  and  lower  levels  of  knowledge  (McCormack,  Ross,  Daugherty,  and 
Garfinkel,  2000;  McCormack,  Garfmkel,  Hibbard,  Keller,  and  Kilpatrick,  in  review).  For  this 
project,  we  look  to  these  indices  as  sources  of  questions  for  future  rounds  of  the  MCBS.  There  is, 
however,  one  other  MCBS  knowledge  measure,  the  8-item  quiz  appearing  for  the  first  time  in 
round  23  of  the  1998  MCBS',  that  was  found  to  have  the  best  psychometric  properties  of  the 
MCBS  measures  of  demonstrated  knowledge^and  is  therefore  used  here  for  comparison  with  the 
Kansas  City  and  National  Evaluation  measures. 

Two  sets  of  psychometric  analyses  were  conducted:  a)  scale-level  analyses  and  b)  item- 
level  analyses.  The  scale-level  analyses  were  designed  to  determine  whether  any  of  the  existing 
scales  by  themselves  could  provide  the  set  of  core  knowledge  questions  for  fiiture  rounds  of  the 
MCBS.  For  these  analyses,  the  knowledge  indices  from  the  Kansas  City  and  National 
Evaluation  were  assessed  according  to  their  reliability  and  construct  validity  and  compared  to  the 
8-item  quiz  from  the  1998  MCBS.  In  addition,  factor  analyses  were  conducted  to  examine  the 
possibility  of  creating  sub-scales.  In  contrast  to  the  scale-level  analyses,  the  item-level  analyses 
examined  the  knowledge  questions  individually  to  determine  which  items  might  be  useftil  for 
ftiture  rounds  of  the  MCBS.  For  these  analyses,  the  individual  knowledge  questions  were 
evaluated  with  respect  to  their  difficulty,  number  of  response  options,  and  percentage  of  don't 
know  responses. 


'  This  8-item  quiz  is  introduced  here  for  the  first  time  because  no  comparable  items  were  included  in  earlier  years  of 
the  MCBS  and  thus  it  could  not  be  considered  as  part  of  a  pre-NMEP  baseline  period. 

^  Note  that  the  MCBS  perceived  Medicare  knowledge  index  also  had  good  psychometric  properties  but  it  is  not 
presented  here  because  it  is  a  measure  of  perceived  knowledge,  see  Bann  et  al.,  2000. 


23 


6.1.1  Scale-Level  Analyses 


6.1.1.1  Reliability 

The  reliability  of  the  scales  was  estimated  using  the  internal  consistency  method  which 
assesses  the  degree  to  which  the  items  on  a  scale  are  related  to  each  other  and  therefore  appear  to 
be  measuring  the  same  construct.  Internal  consistency  reliability  was  computed  using 
Cronbach's  coefficient  alpha  (Cronbach,  1 95 1 ).  Table  15  illustrates  how  the  three  indices 
compare  in  terms  of  reliability.  The  National  Evaluation  index  demonstrated  the  greatest  internal 
consistency.  However,  it  should  be  noted  that  the  National  Evaluation  scale  has  the  largest 
number  of  items  of  the  three  scales  and  all  things  being  equal,  reliability  will  improve  as  the 
number  of  items  is  increased. 

Therefore,  in  order  to  "level  the  playing  field,"  we  used  the  Spearman-Brown  prophecy 
formula  to  project  the  coefficient  alphas  for  the  other  two  scales  if  they  were  to  be  increased  to 
23  items,  the  same  number  of  items  as  the  National  Evaluation  index  (Nunnally  &  Bernstein, 
1994).  Using  this  formula,  the  alpha  for  the  Kansas  City  index  would  increase  slightly  to  0.74, 
however,  the  alpha  for  the  MCBS  would  increase  dramatically  to  a  value  of  0.91.  Comparing 
these  projected  alphas  to  the  value  of  alpha  for  the  National  Evaluation  index  indicates  that  the 
MCBS  quiz  is  the  most  homogeneous  of  the  three  scales.  Therefore,  the  greatest  reliability 
relative  to  number  of  items  would  be  achieved  by  adding  like  items  to  the  MCBS  quiz.  The 
comparatively  lower  homogeneity  of  the  National  Evaluation  and  Kansas  City  indices  is  also 
evidenced  in  the  results  of  the  factor  analysis  described  below  which  suggests  that  these  indices 
contain  subscales. 


Table  15.  Reliability  of  Various  Knowledge  Indices 


Kansas  City  IS-item 
Index 

National  Evaluation  23- 
item  Index 

MCBS  Round  23 
8-item  Quiz 

Cronbach's  Alpha 

0.72 

0.81 

0.77 

6.1.1.2    Construct  Validity 


Evidence  for  the  construct  validity  of  an  index  score  is  obtained  by  observing  how  that 
score  is  associated  with  separate  measures  of  the  same  attribute.  A  high  correlation  between  the 
score  and  other  measures  of  the  same  attribute  is  evidence  of  validity.  Using  general  linear 
models,  we  evaluated  the  validity  of  the  indices  by  comparing  index  scores  with  self-reported 
measures  of  knowledge.  The  following  question  was  used  to  measure  self-reported 
understanding  for  both  the  Kansas  City  and  the  National  evaluation: 

"How  would  you  rate  your  understanding  of  the  different  types  of  health  insurance 
options  for  people  with  Medicare?  "  (Poor/Fair/Good/Very  good/Excellent) 

In  the  Kansas  City  data,  self-rated  understanding  of  Medicare  was  positively  related  to 
the  knowledge  index  (p<0.0001)  and  the  ordering  of  means  by  self-rated  level  of  knowledge  was 
monotonic.  Duncan's  Multiple  Range  test  (Duncan,  1975)  detected  significant  differences 
between  the  mean  index  scores  at  each  level  of  self-rated  understanding,  with  one  exception: 
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there  was  not  a  significant  difference  in  index  scores  between  those  who  rated  their 
understanding  "very  good"  and  "excellent,"  although  the  mean  index  score  for  those  in  the 
excellent  group  was  higher. 

Using  the  National  Evaluation  data,  self-rated  understanding  was  positively  related  to  the 
knowledge  index  (p  <  0.0001),  and  the  ordering  of  the  means  by  self-rated  knowledge  was  also 
monotonic  with  one  exception:  those  respondents  who  identified  themselves  as  having  an 
excellent  understanding  of  Medicare  had  a  lower  average  index  score  than  those  who  identified 
themselves  as  having  a  very  good  understanding.  A  Duncan's  multiple  range  test  found  that  the 
mean  knowledge  index  was  significantly  higher  for  those  whose  self-reported  knowledge  was 
fair  than  for  those  who  reported  it  was  poor,  for  good  compared  to  fair,  and  for  very  good 
compared  to  good,  but  excellent  was  significantly  lower  than  very  good  and  not  significantly 
different  from  fair  to  good. 

We  used  the  following  question  from  the  1998  MCBS  Round  23  to  evaluate  the  validity 
of  the  8-item  quiz: 

"How  much  do  you  think  you  know  about  the  Medicare  program?  "  (Just  about 
everything/Most/Some/A  little/Almost  none) 

The  ordering  of  the  means  was  monotonic  with  the  exception  of  the  last  two  categories, 
"most"  and  "just  about  everything."  The  Duncan's  multiple  range  test  indicated  no  significant 
difference  in  the  8-item  quiz  scores  between  these  two  levels.  There  were  significant  differences 
between  all  other  levels. 

Overall,  these  associations  support  the  construct  validity  of  all  three  indices  equally  and 
do  not  suggest  that  one  index  is  better  than  another.  An  interesting  result  from  these  analyses  is 
that  the  highest  response  choices  (e.g.,  "excellent"  and  "just  about  everything")  do  not  appear  to 
be  behaving  as  would  be  expected.  We  suggest  cognitive  testing  of  these  response  scales  to 
determine  how  the  higher  response  choices  are  being  interpreted  by  respondents. 

6.1.1.3     Factor  Analysis. 

Factor  analysis  may  be  used  to  provide  additional  evidence  for  the  construct  validity  of  a 
scale.  Examining  the  clustering  of  the  items  into  factors  provides  information  about  the 
underlying  constructs  being  measured  by  the  scale.  Factor  analysis  results  may  also  be  used  to 
determine  if  the  scale  can  be  divided  into  separate  sub-scales. 

Principal  factor  analyses  with  promax  rotation  were  conducted  on  each  of  the  three 
knowledge  indices.  For  the  Kansas  City  index,  a  three-factor  solution  was  appropriate.  Table  16 
contains  the  factor  loadings,  with  loadings  greater  than  0.30  in  bold.  The  first  factor  reflects 
beneficiaries'  knowledge  of  the  Original  Medicare  program  which  included  questions  6-9  and 
15-17.  The  second  factor  reflects  awareness  of  the  different  Medicare  insurance  options  and  the 
third  factor  reflects  knowledge  of  Medicare  managed  care. 
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Table  16.  Factor  Loadings  for  the  Three  Factor  Solution  of  the  Kansas  City  Knowledge  Index 


Iteni# 

Description 

Factor  1 : 
Original 
jYiCQicare 

Factor  2: 
Awareness 

Ul  inaliralicc 

r  actor  o: 
Medicare 

Ca  re 

6 

As  far  as  you  know,  can  people  on  Medicare  still  get  Original  Medicare  today/ 

n  dA 
11. t** 

0  17 

7 

As  far  as  you  know,  does  Original  Medicare  pay  tor  all  nealtn  care  costs  tor  peopie  on 
Medicare? 

0. 1 1 

0.04 

8 

As  far  as  you  know,  are  there  different  types  of  private  health  insurance  plans  that  people  on 
Medicare  can  get  to  pay  tor  tnings  mat  ivieaicare  uoesn  i  pay  lui : 

0.54 

0.06 

0.09 

9 

Have  you  ever  heard  of  a  Medicare  supplemental  health  insurance  plan,  sometimes  called  a 
Medigap  plan? 

0.20 

0.40 

0.07 

10 

Have  you  ever  heard  of  a  Medicare  HMO? 

-0.20 

0.20 

0.90 

11 

Have  you  ever  heard  of  a  Medicare  managed  care  plan? 

-0.06 

0.64 

0.17 

12 

Are  there  Medicare  [HMO/managed  care  plans]  in  the  Kansas  City  area? 

0.07 

-0.17 

0.76 

13 

Have  you  ever  heard  of  an  MSA  or  medical  savings  account  for  people  on  Medicare? 

0.11 

0.87 

-0.10 

15 

If  a  person  signs  up  for  any  of  these  Medicare  health  insurance  plans,  is  he  or  she  still  in  the 
Medicare  program? 

A  CO 

-U.U  1 

n  HQ 

16 

If  a  person  signs  up  for  any  of  these  Medicare  health  insurance  plans,  does  he  or  she  still  get 
at  least  the  same  health  care  benefits  as  someone  in  the  Original  Medicare  plan? 

u.oo 

-0  OR 

0  01 

17 

Do  people  who  are  happy  with  the  health  insurance  plan  they  have  now  have  to  change 
plans  this  year? 

0.52 

0.14 

0.04 

19 

In  this  type  of  Medicare  health  insurance  plan,  a  person  needs  to  go  to  doctors  and  hospitals 
from  a  list  given  bv  the  olan.  Does  this  statement  best  describe... (Original  Medicare/A 
Medicare  HMO/Both/Neither) 

0.18 

0.00 

0.63 

21 

Which  type  of  Medicare  health  insurance  plan  is  more  likely  to  pay  for  most  prescription 
drugs''  Would  you  say... (Original  Medicare/A  Medicare  HMO/Both/Neither) 

0.28 

0.06 

0.43 

23 

Which  type  of  Medicare  health  insurance  plan  only  allows  a  person  to  leave  the  plan  and 
change  to  another  plan  once  a  year?  Would  you  say . .  .(A  Medicare  HMO/An 
MSA/Both/Neither) 

0.03 

0.82 

-0.06 

35 

Is  there  a  local  service  in  the  Kansas  City  area  that  offers  free  and  unbiased  counseling  to 
people  on  Medicare  about  choosing  a  health  insurance  plan? 

0.19 

0.02 

0.04 

Next,  we  used  the  results  of  the  factor  analysis  to  create  three  sub-scales,  representing  the 
three  factors  (Original  Medicare,  Awareness  of  Insurance  Options,  and  Medicare  Managed 
Care).  The  sub-scales  were  created  by  unit-weighting  and  summing  the  items  with  factor 
loadings  greater  than  0.30.  We  then  calculated  coefficient  alphas  for  the  sub-scales  to  assess 
their  reliability  and  determine  if  any  sub-scale  by  itself  reached  an  acceptable  level  of  reliability 
(See  Table  1 7).  The  third  factor  —  Medicare  managed  care  —  had  the  highest  alpha  of  0.62, 
with  the  others  being  0.52  and  0.58. 


Table  17.  Alphas  for  Sub-scales:  Kansas  City  Knowledge  Index 


Factor 

Description  of  Factor 

Questions  Comprising  Factor 

Coefficient  Alpha 

1 

Original  Medicare 

6,7,8,  15,  16,  17 

0.58 

2 

Awareness  of  Insurance  Options 

9,  11,  13,23 

0.52 

3 

Medicare  Managed  Care 

10,  12,  19,21 

0.62 

For  the  National  Evaluation  index,  five  factors  were  noted:  Awareness  of  Medicare 
insurance  options,  knowledge  of  Original  Medicare,  awareness  of  information  sources  and  other 
resources,  knowledge  of  Medicare  managed  care,  and  knowledge  of  Medigap  insurance  (See 
Table  18). 


27 


Table  18.  Factor  Loadings  for  Five  Factor  Solution  of  National  Evaluation  Knowledge  Index* 


Item  # 

Liescripiion 

Factor  1 : 
Awareness 
of 

Insurance 
Options 

Factor  2: 
Original 
Medicare 

Factor  3: 
Information 
and 
Resources 

Factor  4: 
Medicare 
Managed 
Care 

Factor  5: 
Medigap 
insurance 

J 

Ac  fat-  -ao       irmu/  nrp  thprf*  Hiffprpnt  tvne<i  of  health  insurance  ODtions  that  people  with 
Medicare  can  get? 

0.78 

-0.01 

0.04 

-0.03 

-0.08 

A 

Hax/p  vnii  pvpr  hfarH  of  a  Medicare  sunDlemental  health  insurance  plan,  which  is  also 
sometimes  called  a  "Medigap"  policy? 

0.59 

0.07 

0.00 

-0.09 

0.23 

C 

Hax/p  \inii  pvpr  hfarH  nf  a  MpHlrare  manaeed  care  olan  for  example,  a  health  maintenance 
organization  or  HMO? 

0.78 

0.05 

-0.05 

0.01 

-0.06 

7 

If  a  person  signs  up  for  any  of  these  -  Medigap,  a  Medicare  Managed  Care  Plan,  or  an 
inrliicivp  PrnviHpr  Orpanization  —  do  thev  still  have  Medicare? 

0.62 

-0.11 

0.10 

0.05 

0.21 

8 

As  far  as  you  know,  can  you  still  get  the  Original  Medicare  plan  today? 

0.25 

0.36 

0.12 

0.11 

-0.12 

9 

The  Medicare  program  has  recently  begun  to  offer  more  information  and  help  in  order  to 

answer  QUeSllOnS  auOUl  Uic  IvicuiCaic  piugiaiii. 

0.18 

0.12 

0.45 

0.03 

-0.05 

10 

As  far  as  you  know,  will  out-of-pocket  costs  differ  dependmg  on  which  health  msurance 
option  a  person  signs  up  for? 

0.49 

0.08 

0.06 

0.06 

0.06 

1 1 

iiru"  u     „  »^^Un  A-^timirin/T    of0rv«onf c  ic  trilp  nhmit  what  thp  Oripinal  Medicare  olan  oavs 
Which  one  oi  the  following  statements  is  irue  aooui  wimi  mc     igmai  ivn^uivai v  piun  y^j-^ 

for?  It  pays  for...  (All  health  care  costs/Some  health  care  costs/Only  emergency  health 
care/Only  prescription  drugs) 

0.28 

0.30 

-0.17 

-0.09 

0.26 

12 

11.,^      «  XAaAlr^nri^  \Ar%nnnaA  ^^arp  Plan  nnH  wantpH  tf>  leave  it  which  One  Ot 

If  you  were  enrolled  in  a  Medicare  ivianagea  uare  rian  anu  wdnicu  lu  icavc  u,  vyiih,ii  wn..  i^. 
the  following  statements  is  true?  You  can  ...  (Leave  any  time  but  won't  be  covered  by 
Medicare/Leave  unuer  special  circumsiances/LCdvc  any  mnc; 

0.34 

0.10 

0.09 

0.32 

-0.04 

13 

Which  one  of  the  following  is  a  reason  for  contacting  your  state's  Peer  Review  Organization 
(PRO)?  (Find  support  group  of  peers/Complaint  about  quality  of  Medicare  care/Change 
health  insurance  option/iNone  or  me  aoove; 

0.02 

0.17 

0.19 

0.18 

0.32 

14 

Which  one  of  the  following  is  likely  to  happen  when  you  go  to  a  doctor  who  accepts 
assignment?"  (Save  money/Pay  more  money/Save  money  only  on  services  not  normally 
covered)   

0.04 

0.50 

0.04 

0.05 

0.08 

15 

Which  one  of  the  following  is  paid  for  if  you  only  have  the  Original  Medicare  Plan?  (Yearly 
dental  exams/Pneumonia  shot/Medical  care  outside  US/Long-term  care) 

-0.01 

0.52 

-0.08 

-0.03 

0.24 

16 

Which  type  of  health  insurance  option  gives  you  more  freedom  to  choose  the  doctors  or 
hospitals  you  want  to  go  to?  (Original  Medicare  plan/Medicare  managed  care  plan/Inclusive 
Provider  organization/all  about  the  same) 

-0.04 

0.67 

0.08 

-0.03 

-0.02 

17 

Which  type  of  health  insurance  option  is  least  likely  to  pay  for  prescription  drugs.'  (Original 
Medicare/Medicare  managed  care  plan/Medigap/AII  about  the  same) 

0.08 

0.59 

-0.02 

-0.01 

0.11 

1  Si 
1  5 

wViinh  fvnp  nf  hpalth  insurance  ontion  can  refuse  to  sell  vou  a  Dolicv  after  age  65  and  a  halt 
because  of  poor  health?  (Original  Medicare/Medicare  managed  care  plan/Medigap/None  of 
the  above) 

0.07 

0.10 

0.03 

-0.07 

0.51 

1Q 

Which  type  of  health  insurance  option  will  cover  a  6-month  stay  in  a  nursing  home? 
(Original  Medicare/Medicare  managed  care  plan/Both/Neither) 

-0.04 

0.22 

-0.03 

0.07 

0.36 

20 

Emergency  health  care  is  paid  for  wherever  you  are  in  the  US  under  which  type  of  insurance 
option?  (Original  Medicare/Medicare  managed  care  plan/Both/Neither) 

0.07 

0.00 

-0.13 

0.72 

0.11 

21 

The  following  statement  is  true  for  which  type  of  health  insurance  option:  "Medicare  gives 
you  the  right  to  appeal  decisions  about  what  a  Medicare  plan  pays  for?"  (Original 
Medicare/Medicare  managed  care  plan/Both/Neither) 

0.01 

-0.07 

-0.01 

0.69 

0.21 

The  Original  Medicare  Plan  now  pays  for  more  preventive  health  care  services  like  breast  x- 
rays  and  diabetes  monitoring.  Is  this  also  true  for  all,  some,  or  none  of  the  other  different 
types  of  insurance  options? 

-0.07 

0.03 

0.14 

0.54 

-0.21 

23 

If  you  had  a  Medigap  policy  and  dropped  it,  when  could  you  get  it  back?  (Any  time/Only 
under  some  conditions/Never) 

-0.02 

0.04 

0.33 

0.02 

0.51 

39 

Is  there  an  information  and  counseling  service  that  people  with  Medicare  can  use  to  get  help 
understanding  and  comparing  health  insurance  options? 

0.09 

-0.04 

0.67 

0.01 

0.04 

41 

Does  the  Medicare  program  have  its  own  Internet  website? 

-0.07 

-0.03 

0.71 

-0.03 

0.10 

Notes: 

*  Excludes  question  six  which  asked  about  awareness  of  a  fictitious  plan.  This  question  was  used  to  assess  potential  social  desirability 
bias. 


The  five  factors  had  alphas  ranging  from  0.48  to  0.72  (See  Table  19).  The  Original 
Medicare  sub-scale  had  an  alpha  of  0.72,  which  is  fairly  high,  particularly  for  a  five-item  scale. 
The  next  highest  alpha  was  for  the  awareness  of  insurance  options  sub-scale.  These  two  sub- 
scales  address  basic  knowledge  of  the  Medicare  systems  (i.e..  Original  Medicare  and  Awareness 
of  Insurance  Options)  and  achieve  good  reliability  for  the  small  number  of  items  comprising 
them.  Therefore,  these  questions  may  be  leading  contenders  for  a  set  of  core  MCBS  items. 


Table  19.  Alphas  for  Sub-scales:  National  Evaluation  Knowledge  Index 


Factor 

Description  of  Factor 

Questions  Comprising  Factor 

Coefficient  Alpha 

1 

Awareness  of  Insurance  Options 

8,  11,  14,  15,  16,  17 

0.64 

2 

Original  Medicare 

3,4,5,7,  10,  12 

0.72 

3 

Information  and  Resources 

9,  23,39,41 

0.55 

4 

Managed  Care 

12,  20,21,22 

0.55 

5 

Medigap  Insurance 

13,  18,  19,23 

0.48 

The  factor  analysis  of  the  1998  MCBS  eight-item  quiz  (Round  23)  revealed  that  the  quiz 
items  compose  only  one  factor  (See  Table  20).  This  suggests  that  the  items  are  all  highly  related 
and  appear  to  be  measuring  the  same  underlying  construct.  Because  the  items  form  one  factor, 
no  sub-scales  were  created. 


The  results  of  the  scale-level  analyses  suggest  that  one  useful  source  of  knowledge 
questions  may  be  the  National  Evaluation  index.  The  coefficient  alphas  suggest  that  the  National 
Evaluation  index  had  the  highest  reliability  (although  it  should  be  noted  that  this  index  also  had 
the  largest  number  of  items).  The  results  of  the  factor  analysis  indicate  that  the  National 
Evaluation  index  had  the  greatest  number  of  factors  and  therefore  covered  the  widest  variety  of 
topics  among  the  three  scales.  This  suggests  that  this  scale  is  better  able  to  assess  the  variety  of 
information  that  beneficiaries  need  to  understand  the  Medicare  program.  The  National 
Evaluation  index  also  contained  a  sub-scale,  "Original  Medicare,"  which  demonstrated  good 
reliability. 

The  scale-level  analyses  allowed  us  to  examine  the  properties  of  each  scale  as  a  whole. 
However,  the  results  of  these  analyses  do  not  provide  information  about  the  psychometric 
properties  of  the  individual  items  that  comprise  a  scale.  The  next  step  in  identifying  candidates 
to  be  included  in  future  rounds  of  the  MCBS  is  to  evaluate  the  properties  of  each  item 
individually. 
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Table  20.  Factor  Loadings  for  One  Factor  Solution  of  1998  MCBS  (Round  23)  8-item  Quiz 


Itein# 

Description 

Factor  1: 
Managed 
care 

43 

Most  people  covered  by  Medicare  can  select  among  different  kinds  of  health  plan  options 
within  Medicare. 

0.51 

44 

Medicare  without  a  supplemental  insurance  policy  pays  for  all  of  your  health  care  expenses. 

0.44 

45 

If  you  are  happy  with  the  way  you  currently  receive  health  care,  you  do  not  have  to  make 
any  changes  in  the  way  you  get  your  Medicare  services. 

0.52 

46 

The  Medicare  program  has  recently  begun  to  offer  more  information  and  help  in  order  to 
answer  your  Medicare  questions. 

0.43 

47 

People  can  report  complaints  to  Medicare  about  their  Medicare  managed  care  plans  (HMOs) 
or  supplemental  plans  if  they  are  not  satisfied  with  them. 

U.  JJ 

48 

If  someone  joins  a  Medicare  managed  care  plan  (HMO)  that  covers  people  on  Medicare, 
they  have  limited  choices  about  which  doctors  they  can  see. 

0.62 

49 

If  someone  joins  a  Medicare  managed  care  plan  (HMO)  that  covers  people  on  Medicare, 
they  can  change  or  drop  the  plan  and  still  be  covered  by  Medicare. 

0.64 

50 

Medicare  managed  care  plans  (HMOs)  that  cover  people  on  Medicare  often  cover  more 
health  services,  like  prescribed  medicines  than  Medicare  without  a  supplemental  policy. 

0.58 

6.1.2       Item-Level  Comparisons 


Please  note  that  it  is  not  possible  to  make  direct  comparisons  between  items  in  the 
different  questionnaires  because  different  groups  of  respondents  filled  out  each  questionnaire.  In 
this  case  we  can  not  tell  whether  differences  in  the  item  parameters  are  due  to  differences  in  the 
items  or  differences  in  the  respondents.  For  example,  a  difficult  question  may  seem  easy  if  it  is 
administered  to  respondents  with  a  high  level  of  knowledge  while  it  may  seem  very  difficult  if  it 
is  administered  to  respondents  with  very  little  knowledge.  Therefore,  for  these  analyses,  items 
may  only  be  compared  to  other  items  on  the  same  scale.  For  example,  we  will  determine 
whether  a  particular  item  is  more  difficult  than  other  items  on  the  same  scale.  It  will  be  possible 
to  compare  all  of  the  items  to  each  other  if,  in  the  future,  all  of  these  knowledge  items  are 
administered  to  the  same  sample  of  beneficiaries. 

The  results  of  the  item-level  analyses  are  presented  separately  for  each  index  in  Tables  21 
to  23.  The  number  of  response  options,  the  percent  of  correct  responses,  and  the  percent  of  don't 
know  responses  are  presented  for  each  item. 

6.1.2.1     Item  Difficulty 

An  important  characteristic  of  an  item  designed  to  measure  knowledge  is  the  difficulty  of 
the  item.  One  indicator  of  an  item's  difficulty  is  the  proportion  of  respondents  who  correctly 
answered  the  item.  Items  with  greater  difficulty  will  have  a  lower  proportion  of  correct 
responses  while  easier  items  will  have  more  correct  responses. 

The  percentage  distribution  of  correct  responses  for  the  items  included  in  the  Kansas  City 
evaluation  is  displayed  in  Table  21.  Clearly,  item  23  was  the  most  difficult  question  on  the  scale 
with  only  4  percent  of  respondents  answering  this  item  correctly.  In  contrast,  item  8  was  a  very 
easy  item  with  92  percent  of  respondents  answering  it  correctly.  Overall,  with  the  exception  of 
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item  23,  the  items  on  the  Kansas  City  index  have  difficuhy  levels  ranging  from  moderate  to  easy. 
The  ability  of  the  scale  to  detect  respondents  with  higher  levels  of  knowledge  could  be  enhanced 
by  adding  some  additional  difficult  items  to  the  scale. 

The  percentage  of  correct  responses  for  each  knowledge  question  included  in  the 
National  Evaluation  are  displayed  in  Table  22.  The  most  difficult  item  in  the  National 
evaluation  was  question  1 8  with  only  an  1 1  percent  correct  response  rate.  Item  1 1  was  the 
easiest  knowledge  question  with  82  percent  of  respondents  answering  this  question  correctly. 
Overall,  the  National  Evaluation  contained  questions  with  a  variety  of  difficulty  levels. 

Table  23  displays  the  proportion  of  correct  responses  for  each  of  the  eight  items  included 
in  Round  23  of  the  MCBS.  Of  the  eight  items,  item  44  was  the  easiest  and  item  50  was  the  most 
difficult.  Overall,  the  quiz  contained  items  with  a  medium  level  of  difficulty.  None  of  the  items 
on  the  quiz  were  either  extremely  easy  or  extremely  difficult.  These  items  would  be  useful  for 
measuring  knowledge  of  beneficiaries  with  an  average  level  of  knowledge. 

Ideally,  a  knowledge  measure  should  include  items  with  a  wide  range  of  difficulty  levels. 
This  is  necessary  to  adequately  assess  the  knowledge  of  beneficiaries  at  all  levels  of  ability.  For 
example,  a  scale  containing  only  very  difficult  items  can  only  discriminate  between  respondents 
at  the  very  highest  knowledge  levels.  Conversely,  very  easy  items  are  only  useful  for  measuring 
knowledge  of  respondents  with  very  little  knowledge.  Therefore,  because  the  MCBS  is  designed 
to  assess  knowledge  among  a  diverse  populafion  of  beneficiaries,  future  rounds  of  the  MCBS 
should  include  items  with  a  variety  of  difficulty  levels. 

6.1.2.2     Percentage  of  Don't  Know  Responses. 

For  each  of  the  knowledge  indices,  respondents  were  given  the  option  of  indicating  that 
they  did  not  know  the  answer  to  a  question.  Examining  the  frequency  of  don't  know  responses  is 
useful  for  identifying  items  that  may  be  particularly  confusing  or  difficult  for  respondents.  Table 
21  contains  the  percentage  of  don't  know  responses  for  the  Kansas  City  knowledge  questions. 
The  majority  of  quesUons  had  a  low  percentage  of  don't  know  responses  with  one  exception:  51 
percent  of  respondents  indicated  that  they  did  not  know  the  answer  to  item  35.  This  question 
concerns  the  availability  of  free  counseling  on  health  insurance  plans  in  Kansas  City.  The  high 
percentage  of  don't  know  responses  for  this  question  suggests  that  approximately  one-half  of 
respondents  were  unaware  of  this  service. 

The  proportions  of  don't  know  responses  for  the  National  Evaluation  items  are  presented 
in  Table  22.  Nine  of  the  23  items  included  in  the  National  Evaluation  index  have  don't  know 
response  rates  of  more  than  50  percent.  Generally,  more  difficult  questions  had  a  higher 
percentage  of  don't  know  responses,  suggesting  that  there  was  some  tendency  for  respondents  to 
indicate  that  they  did  not  know  the  answer  to  a  question  rather  than  responding  incorrectly.  The 
percentage  of  don't  know  responses  may  also  suggest  potential  problems  with  an  item.  For 
example,  half  of  the  respondents  indicated  that  they  did  not  know  the  answer  to  item  7.  This 
question  includes  the  name  of  a  fictitious  plan,  the  Inclusive  Provider  Organization,  which  may 
have  confused  respondents.  Another  question,  item  41 ,  asked  whether  the  Medicare  program  has 
its  own  website.  This  question  had  the  highest  percentage  (81%)  of  don't  know  answers.  A 
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possible  explanation  for  this  is  that  many  Medicare  beneficiaries  may  not  have  access  to  the 
Internet  or  have  tried  to  find  a  Medicare  website. 

Several  of  the  MCBS  quiz  questions  have  a  large  percentage  of  don't  know  responses 
(See  Table  23).  For  example,  approximately  one-half  of  the  respondents  indicated  that  they  did 
not  know  the  correct  answer  to  item  50.  This  item  also  had  the  smallest  percentage  of  correct 
responses,  suggesting  that  this  item  may  have  been  particularly  difficult  for  respondents.  In 
contrast  to  the  other  items  on  the  quiz,  this  item  asks  respondents  to  compare  two  types  of 
insurance  plans  (managed  care  and  Medicare  without  a  supplement).  This  task  is  more 
challenging  than  simply  knowing  the  feature  of  one  particular  plan  which  may  explain  the 
greater  difficulty  of  this  question  for  respondents. 

6.1 .2.3     Number  of  Response  Options 

Another  property  of  questionnaire  items  is  the  number  of  response  choices  available  to 
respondents.  If  an  iicm  has  very  few  response  options,  there  is  a  higher  probability  that  a 
respondent  will  choose  the  correct  response  just  by  chance.  For  example,  if  a  question  has  only 
two  response  choices  then  a  respondent  who  does  not  know  the  answer  to  the  question  has  a  50 
percent  chance  of  choosing  the  correct  answer.  However,  if  the  question  has  four  response 
options,  this  probability  is  reduced  to  25  percent. 

Tables  21  to  23  contain  the  number  of  response  options  for  the  items  on  each  of  the 
knowledge  indices.  The  majority  of  items  on  the  Kansas  City  index  have  two  response  choices, 
yes  or  no.  There  are,  however,  three  items  (19,  21,  and  23)  that  have  four  response  options. 
Because  these  items  allow  the  respondents  to  choose  between  four  possible  options  rather  than 
just  two,  there  is  a  lower  probability  of  respondents  simply  guessing  the  correct  response. 

The  items  on  the  National  evaluation  index  have  a  variety  of  response  options.  Four  of 
the  items  (12, 14,  22,  and  23)  have  four  response  options  and  nine  items  (11,13, 15-21)  have 
three  options.  The  remaining  items  have  only  two  response  options. 

All  of  the  items  on  the  MCBS  quiz  have  two  response  options  (true  or  false).  Therefore, 
for  these  items,  there  is  a  higher  probability  that  a  respondent  who  does  not  know  the  answer  will 
select  the  correct  answer  than  for  the  items  on  the  Kansas  City  and  National  Evaluation  scales 
that  contain  3  or  4  response  categories. 

Overall,  the  National  Evaluation  scale  had  the  largest  number  of  items  with  more  than 
two  response  options.  It  is  important  to  note,  however,  that  an  item  will  not  necessarily  be 
improved  by  simply  adding  more  response  choices.  Adding  response  options  that  are  obviously 
incorrect  will  not  necessarily  decrease  the  probability  of  respondents  guessing  the  correct 
answer.  Also,  it  is  important  to  consider  the  topic  area  when  deciding  on  the  number  of  response 
options.  In  some  cases,  two  or  three  response  options  may  be  the  most  appropriate. 
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Table  21.  Item  Characteristics  for  Questions  Included  in  the  Kansas  City  Knowledge  Index 


Item  # 

Description 

%  Correct 

%  Don't 

MVIIVI  TV  S 

Number  of 
Categories 

6 

As  tar  as  you  know,  can  people  on  Meoicare  snii  get  vjriginai  ivicuicdrc  luutiy : 

76% 

21% 

2 

7 

As  tar  as  you  know,  Goes  unginai  ivieuicare  pay  lor  an  ncaiui  caic  cuaia  lui  jjcupic  uu 
Medicare? 

84% 

8% 

2 

8 

As  far  as  you  know,  are  there  different  types  of  private  health  insurance  plans  that  peop  le  on 

i^nv'i^  r^nn  rt c%\  \f\  r\rx\i  fV\r  thinfTc  that  \^**Hi/^nrf  Hrtpcp't  r^3V  Tor'^ 

ivieoicare  can  gei  lo  pay  lor  iningo  iiiai  ivicuicaiv  uucau  i  y<*j  • 

92% 

6% 

2 

9 

Have  you  ever  heard  of  a  Medicare  supplemental  health  insurance  plan,  sometimes  called  a 
Meaigap  plan  / 

78% 

1% 

2 

10 

Have  you  ever  heard  of  a  Medicare  HMO? 

87% 

1% 

2 

1 1 

Have  you  ever  neara  oi  a  Meuicare  managcu  caic  piaii: 

63% 

4% 

2 

12 

Are  there  Medicare  [HMO/managed  care  plans]  in  the  Kansas  City  area? 

57% 

30% 

2 

13 

Have  you  ever  heard  of  an  MSA  or  medical  savings  account  for  people  on  Medicare? 

33% 

2% 

2 

15 

If  a  person  signs  up  for  any  or  these  Medicare  nealtn  insurance  pians,  is  ne  or  bnc  buu  m  mc 
Medicare  program? 

71% 

22% 

2 

16 

If  a  person  signs  up  tor  any  ot  these  Medicare  neaitn  insurance  pians,  uoci.  iic  ui  inc  sun  gci 
at  least  the  same  health  care  benefits  as  someone  in  the  Original  Medicare  plan? 

58% 

31% 

2 

17 

Do  people  who  are  happy  with  the  health  insurance  plan  they  have  now  have  to  change 
plans  this  year? 

76% 

19% 

2 

19 

In  this  type  of  Medicare  health  insurance  plan,  a  person  needs  to  go  to  doctors  and  hospitals 
from  a  list  given  by  the  plan.  Does  this  statement  best  describe... (Original  Medicare/A 
Medicare  HMO/Both/Neither) 

56% 

18% 

4 

21 

Which  type  of  Medicare  health  insurance  plan  is  more  likely  to  pay  for  most  prescription 
drugs'^  Would  you  say... (Original  Medicare/A  Medicare  HMO/Both/Neither) 

42% 

21% 

4 

23 

Which  type  of  Medicare  health  insurance  plan  only  allows  a  person  to  leave  the  plan  and 
change  to  another  plan  once  a  year?  Would  you  say. ..(A  Medicare  HMO/An 
MSA/Both/Neither) 

4% 

14% 

4 

35 

Is  there  a  local  service  in  the  Kansas  City  area  that  offers  free  and  unbiased  counseling  to 
people  on  Medicare  about  choosing  a  health  insurance  plan? 

32% 

51% 

2 

Table  22.  Item  Characteristics  for  Questions  Included  in  National  Evaluation  Knowledge  Index* 


Item  # 

Description 

/o  v-urrcci 

0/.  nnn^t 

Knows 

IViimhpt*  CkT 

Response 
Options 

3 

As  far  as  you  know,  are  there  different  types  oi  health  insurance  options  mat  peopie  wiin 
Medicare  can  get? 

/  \J  /o 

19% 

2 

4 

Have  you  ever  heard  of  a  Medicare  supplemental  health  insurance  plan,  which  is  also 
sometimes  called  a  "Medigap"  policy? 

/  \J  /o 

9% 

2 

5 

Have  you  ever  heard  ot  a  Medicare  managea  care  pian,  lor  exampic,  <x  iicdiui  iiidiiucnaiii-c 
organization  or  HMO? 

70% 

9% 

2 

7 

If  a  person  signs  up  for  any  of  these  -  Medigap,  a  Medicare  Managed  Care  Plan,  or  an 
Inclusive  Provider  Organization  -  do  they  still  have  Medicare? 

45% 

50% 

2 

8 

As  far  as  you  know,  can  you  still  get  the  Original  Medicare  plan  today? 

74% 

25% 

2 

9 

The  Medicare  program  has  recently  begun  to  offer  more  information  and  help  in  order  to 
answer  questions  about  the  Medicare  program. 

64% 

31% 

2 

10 

As  far  as  you  know,  will  out-of-pocket  costs  differ  depending  on  which  health  insurance 
option  a  person  signs  up  for? 

67% 

31% 

2 

11 

Which  one  of  the  following  statements  is  true  about  what  the  Original  Medicare  plan  pays 
for?  It  pays  for...  (All  health  care  costs/Some  health  care  costs/Only  emergency  health 
care/Only  prescription  drugs) 

oz  /o 

1 1% 

1  1  /O 

4 

12 

If  you  were  enrolled  in  a  Medicare  Managed  (Jare  h'lan  ana  wanted  to  leave  it,  wnicn  one  oi 
the  following  statements  is  true?  You  can  ...  (Leave  any  time  but  won't  be  covered  by 
Medicare/Leave  under  special  circumstances/Leave  any  time) 

29% 

60% 

3 

13 

Which  one  of  the  following  is  a  reason  for  contacting  your  state's  Peer  Review  Organization 
(PRO)?  (Find  support  group  of  peers/Complaint  about  quality  of  Medicare  care/Change 
health  insurance  option/None  of  the  above) 

32% 

52% 

4 

14 

Which  one  of  the  following  is  likely  to  happen  when  you  go  to  a  doctor  who  accepts 
assignment?"  (Save  money/Pay  more  money/Save  money  only  on  services  not  normally 
covered) 

63% 

25% 

3 

Wh\rh  nnp  nf  the  followine  is  oaid  for  if  vou  only  have  the  Original  Medicare  Plan?  (Yearly 
dental  exams/Pneumonia  shot/Medical  care  outside  US/Long-term  care) 

51% 

36% 

4 

16 

Which  type  of  health  insurance  option  gives  you  more  freedom  to  choose  the  doctors  or 
hospitals  you  want  to  go  to?  (Original  Medicare  plan/Medicare  managed  care  plan/Inclusive 
Provider  organization/all  about  the  same) 

59% 

25% 

4 

17 

Which  type  of  health  insurance  option  is  least  likely  to  pay  for  prescription  drugs.'  (Unginai 
Medicare/Medicare  managed  care  plan/Medigap/AII  about  the  same) 

52% 

33% 

4 

1 X 

1  o 

Which  type  of  health  insurance  option  can  refuse  to  sell  you  a  policy  after  age  65  and  a  half 
because  of  poor  health?  (Original  Medicare/Medicare  managed  care  pian/Medigap/None  of 
the  above) 

11% 

63% 

4 

Which  type  of  health  insurance  option  will  cover  a  6-month  stay  in  a  nursing  home? 
(Original  Medicare/Medicare  managed  care  plan/Both/Neither) 

20% 

58% 

4 

20 

Emergency  health  care  is  paid  for  wherever  you  are  m  the  US  under  which  type  of  insurance 
option''  (Original  Medicare/Medicare  managed  care  plan/Both/Neither) 

19% 

26% 

4 

21 

The  following  statement  is  true  for  which  type  of  health  insurance  option:  Medicare  gives 
you  the  right  to  appeal  decisions  about  what  a  Medicare  plan  pays  for?"  (Original 
Medicare/Medicare  managed  care  plan/Both/Neither) 

19% 

32% 

4 

22 

The  Original  Medicare  Plan  now  pays  for  more  preventive  health  care  services  like  breast  x- 
rays  and  diabetes  monitoring.  Is  this  also  true  for  all,  some,  or  none  of  the  other  different 
types  of  insurance  options? 

14% 

49% 

3 

23 

If  you  had  a  Medigap  policy  and  dropped  it,  when  could  you  get  it  back?  (Any  time/Only 
under  some  conditions/Never) 

20% 

70% 

3 

39 

Is  there  an  information  and  counseling  service  that  people  with  Medicare  can  use  to  get  help 
understanding  and  comparing  health  insurance  options? 

31% 

54% 

2 

41 

Does  the  Medicare  program  have  its  own  Internet  website? 

16% 

81% 

2 

*  Excludes  question  six  which  asked  about  awareness  of  a  fictitious  plan.  This  question  was  used  to  assess  potential  social  desirability  bias. 


Table  23.  Item  Characteristics  for  Knowledge  Questions  Included  in  the  1998  MCBS  (Round  23)  8-item  Quiz 


Itein# 

Description  , 

%  Correct 

%  Don't 
ivnowa 

Number  of 
Options 

43 

Most  people  covered  by  Medicare  can  select  among  different  kinds  of  health  plan  options 
within  Medicare. 

/no/ 
4  /  /o 

hA  /o 

L 

44 

Medicare  without  a  supplemental  insurance  policy  for  all  of  your  health  care  expenses. 

1  o  /o 

L 

45 

If  you  are  happy  with  the  way  you  currently  receive  health  care,  you  do  not  have  to  make 
any  changes  in  the  way  you  get  your  Medicare  services. 

72% 

23% 

2 

46 

The  Medicare  program  has  recently  begun  to  offer  more  information  and  help  in  order  to 
answer  your  ivicuiLaic  i^uc^iiuiia. 

45% 

45% 

2 

47 

People  can  report  complaints  to  Medicare  about  their  Medicare  managed  care  plans  (HMOs) 
or  supplemental  plans  if  they  are  not  satisfied  with  them. 

58% 

35% 

2 

48 

If  someone  joins  a  Medicare  managed  care  plan  (HMO)  that  covers  people  on  Medicare, 
they  have  limited  choices  about  which  doctors  they  can  see. 

62% 

31% 

2 

49 

If  someone  joins  a  Medicare  managed  care  plan  (HMO)  that  covers  people  on  Medicare, 
they  can  change  or  drop  the  plan  and  still  be  covered  by  Medicare. 

47% 

46% 

2 

50 

Medicare  managed  care  plans  (HMOs)  that  cover  people  on  Medicare  often  cover  more 
health  services,  like  prescribed  medicines  than  Medicare  without  a  supplemental  policy. 

40% 

49% 

2 

6.1 .2.4     Summary  of  Findings  and  implications  for  Future  Research 

Both  the  scale-level  and  item-level  analyses  provided  some  insight  regarding  which 
questions  might  serve  as  a  pool  of  questions  for  measuring  beneficiary  knowledge  in  future 
rounds  of  the  MCBS.  Table  24  provides  a  summary  of  the  key  findings  from  the  comparisons  of 
the  three  indices. 


Table  24.  Summary  of  Findings  from  Psychometric  Analysis 


Kansas  City  1999 
Medicare  &  You 

National  Medicare  & 
You  2000 

1998  MCBS  Round  23 
8-item  quiz 

Number  of  Items  in 
Index 

15 

23 

8 

1.  Scale-Level  Analysis 

Reliability 

Highest  internal 
consistency 

Construct  Validity 

All  scales  found  to  have  comparable  degree  of  construct  validity. 

Factor  Analysis 

3  factors  found  with 
alphas  ranging  from 
0.52  to  0.62 

5  factors  found  with 
alphas  ranging  from 
0.48  to  0.70 

1  factor  found  with 
alpha  of  0.77 

2.  Item-Level  Comparisons 

Item  Difficulty 

Moderate  to  easy 
difTicuity  levels 

%  correct  =  4%  (item 
23)  -92%  (item  8) 

Variety  of  difficulty 
levels 

%  correct  =  11%  (item 
18) -82%  (item  11) 

Moderate  level  of 
difficulty 

%  correct  =40%  (item 
50)  -  75%  (item  43) 

Percent  of  Don't  Know 
(DK)  Responses 

%DK=  1%  (items  9 
and  10)  to  51%  (item 
35) 

%  DK  =  9%  (items  4 
and  5)  to  70%  (item  23) 

%DK=  18%  (item  44) 
to  49%  (item  50) 

Number  of  Response 
Options 

2  responses  =  12 

3  responses  =  0 

4  responses  =  3 

2  responses  =  9 

3  responses  =  4 

4  responses  =  9 

2  responses  =  8 

3  responses  =  0 

4  responses  =  0 

Based  on  the  results  of  the  scale-level  and  item-level  analyses,  the  National  Evaluation 
index  performed  better  than  the  knowledge  questions  from  the  MCBS.  The  National  Evaluation 
index  met  several  of  the  criteria  evaluated  in  this  report.  The  index  had  the  highest  reliability  of 
the  three  scales  and  demonstrated  evidence  of  construct  validity.  The  index  also  contains  a 
variety  of  difficulty  levels,  allowing  it  to  measure  the  knowledge  of  beneficiaries  with  a  wide 
range  of  ability  levels.  Finally,  the  National  Evaluation  index  also  performed  well  with  respect 
to  item  content.  The  results  of  the  factor  analyses  as  well  as  the  content  of  the  items  suggest  that 
this  scale  measures  a  wider  range  of  topics  in  comparison  to  the  other  two  knowledge  scales. 
This  diversity  allows  the  scale  to  more  accurately  assess  the  knowledge  beneficiaries  need  to 
successfully  navigate  the  Medicare  program. 

Nineteen  of  the  items  (questions  3-5,  7,  8,  10-14, 16-18,  20, 21, 23,  39,  and  41)  on  the 
National  Evaluation  index  address  basic  concepts  that  beneficiaries  need  to  know  to  navigate  the 
Medicare  program.  These  topics  tend  to  remain  constant  from  year  to  year.  Therefore,  these 
items  may  be  considered  core  knowledge  questions  that  could  be  administered  each  year.  The 
remaining  items  (questions  9, 15,  19,  and  22)  address  concepts,  such  as  benefits,  that  could 
possibly  change  (although  some  changes  are  less  likely  than  others).  These  questions  may  be 
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considered  supplemental  items  that  could  be  administered  periodically.  Ultimately,  the  value  of 
each  question  would  need  to  be  evaluated  given  recent  legislative  and  policy  changes. 

The  MCBS  8-item  quiz  and  the  Kansas  City  15 -item  knowledge  index  also  contain 
potential  candidate  items  for  inclusion  in  future  rounds  of  the  MCBS.  In  particular,  item  16  on 
the  Kansas  City  index  asks  whether  a  beneficiary  will  receive  the  same  benefits  with  other  health 
plans  as  he/she  would  on  Original  Medicare.  This  question  addresses  a  basic  Medicare  concept 
and  may  be  a  good  candidate  as  a  core  knowledge  item.  Most  of  the  other  items  on  the  Kansas 
City  index  overlap  in  item  content  with  items  on  the  National  Evaluation  index.  For  example, 
item  21  on  the  Kansas  City  index  asks  which  plan  is  more  likely  to  pay  for  prescription  drugs 
while  item  17  on  the  National  Evaluation  index  asks  which  plan  is  least  likely  to  pay  for 
prescription  drugs.  These  items  could  be  interchanged  on  the  two  surveys.  In  this  case,  the  best 
item  of  the  two  might  be  determined  by  desired  difficulty  level  (see  Tables  21  and  22). 

Items  47  and  50  from  the  MCBS  8-item  quiz  may  also  be  useful  core  knowledge 
questions.  Both  of  these  items  demonstrated  a  moderate  level  of  difficulty  (although  item  47  is 
easier  than  item  50).  Item  47  concerns  whether  beneficiaries  can  report  complaints  about  their 
managed  care  plans  while  item  50  compares  the  coverage  of  managed  care  plans  and  Original 
Medicare.  These  two  questions  address  basic  concepts  related  to  managed  care  and  are  good 
candidates  to  be  added  to  the  pool  of  core  knowledge  items.  Item  45  from  the  MCBS  8-item 
quiz  (which  is  essentially  the  same  as  item  17  in  the  Kansas  City  index)  should  probably  be 
dropped  from  consideration  as  this  is  no  longer  one  of  HCFA's  key  messages  to  beneficiaries 
because  of  the  number  of  Medicare  HMOs  that  have  withdrawn  from  the  market. 

Several  possible  modifications  could  improve  the  National  Evaluation  index. 
First,  the  large  percentage  of  don't  know  responses  for  question  7  suggests  that  it  may  be 
confiising  for  respondents.  This  item  contains  the  name  of  a  fictitious  health  plan,  the  Inclusive 
Provider  Organization.  Removing  this  plan  from  the  question  should  make  it  easier  for 
respondents  to  understand.  Similarly,  question  6  asks  whether  the  respondent  has  heard  of  the 
fictitious  plan;  this  question  was  designed  to  measure  social  desirability  bias.  This  item  could  be 
removed  and  replaced  with  another  knowledge  question.  Also,  item  16  contains  the  fictitious 
plan  as  a  response  option  and  that  option  should  be  removed. 

Examining  the  item-level  analyses  for  the  National  Evaluation  index  revealed  that  item 
22  was  also  one  of  the  most  difficult  items  on  the  scale  with  only  14  percent  of  respondents 
providing  the  correct  response  and  approximately  half  of  respondents  indicating  that  they  did  not 
know  the  answer  to  the  question.  Item  22  addresses  coverage  of  preventive  health  care  services 
and  therefore  may  be  an  important  item  to  include  on  the  knowledge  scale.  A  potential  source  of 
the  high  don't  know  response  rate  for  this  question  may  be  due  to  the  response  categories  that 
refer  to  "other  different  types  of  insurance  options."  The  question  does  not  specify  to  which 
particular  types  of  insurance  options  it  refers,  potentially  causing  some  respondents  to  react  to 
this  question  with  uncertainty  that  they  know  the  benefits  of  all  possible  insurance  options.  The 
question  could  be  improved  by  specifying  particular  types  of  insurance  plans  in  the  response 
options. 
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If  the  intention  of  the  knowledge  index  is  to  measure  understanding  of  the  Medicare 
program,  it  is  suggested  that  the  two  awareness  questions,  items  4  and  5,  be  removed  from  the 
index.  These  items  do  not  specifically  address  understanding  and  they  contain  yes/no  response 
options  which  may  be  affected  by  a  tendency  among  some  respondents  to  exhibit  an 
acquiescence  response  bias  (Nunnally  &  Bernstein,  1 994).  These  items  could  then  be  replaced 
with  other  items  that  require  respondents  to  demonstrate  their  knowledge.  However,  the  new 
replacement  items  should  also  have  a  relatively  easy  level  of  difficulty  to  keep  the  distribution  of 
item  difficulty  levels  the  same. 

Finally,  it  is  critical  that  if  the  knowledge  index  in  the  MCBS  will  be  used  to  monitor  the 
Understanding/Knowledge  goal  of  the  NMEP,  that  the  questions  directly  reflect  the  content  of 
the  version  of  the  Medicare  &  You  handbook  distributed  to  beneficiaries  just  prior  to  fielding  the 
MCBS.  As  the  content  and  focus  of  Medicare  &  You  changes,  so  too  must  the  knowledge  index 
questions.  It  is  reasonable  to  assume  that  if  an  issue  is  obscurely  mentioned  in  the  handbook, 
that  many  beneficiaries  may  miss  it.  All  main  messages  or  critical  knowledge  issues  should  be 
repeated  more  than  once  in  the  handbook  in  different  locations.  This  type  of  layering  approach 
has  been  found  to  be  successfiil  in  educating  beneficiaries  (McCormack,  et  al,  1996).  Many 
beneficiaries  still  have  a  rudimentary  understanding  of  the  Medicare  program.  They  must  first 
understand  the  basics  about  Medicare  benefits  and  coverage  before  being  expected  to  compare 
various  options.  Thus,  it  may  be  wise  to  focus  on  educating  beneficiaries  about  the  basics  in  the 
short  run,  with  increased  diversity  of  knowledge  in  the  longer  term. 

6.2     Comparison  of  Responses  to  Questions  Found  in  Both  the  MCBS  and  the 
Medicare  &  You  2000  Evaluation  Instrument 

6.2.1  Comparison  of  Questions  About  Use  of  the  Handbook 

We  compared  responses  to  questions  on  the  use  of  the  Medicare  &  You  handbook  drawn 
from  both  the  1998  MCBS  (Round  23)  and  the  National  Evaluation  survey  to  assess  how 
respondents  answered  questions  that  were  similar  across  the  two  surveys.  Round  23  of  the 
MCBS  contained  5  questions  regarding  use  of  the  handbook  while  the  National  Evaluation 
survey  contained  16  questions  on  this  topic.  Only  two  questions  are  compatible  for  comparison 
as  shown  in  Tables  25  and  26.  We  compared  a  sub-group  of  MCBS  respondents  in  the  five 
states  who  received  the  pilot  demonstration  version  of  the  Medicare  &  You  handbook  to  the 
national  sample  of  respondents  from  the  National  Evaluation  survey.  Persons  responding  in  the 
Medicare  &  You  evaluation  are  persons  in  the  treatment  group  only  of  that  survey. 


Table  25.  Comparison  of  How  Much  of  the  Handbook  Was  Read 


Survey 

Item  # 

Question 

Response  Analyzed 

Frequency 

MCBS 

BK30 

Would  you  say  you  have  read  (it). .  .? 

Thoroughly  or  part  of  it 

63% 

M&Y  2000 

Treat.65 

How  much  of  (it)  did  you  read? 

Most  or  all  or  parts  of  it 

74% 

A  greater  percentage  of  respondents  in  the  Medicare  &  You  survey  indicated  that  they  had  read 
parts  or  all  of  the  handbook  than  respondents  in  the  MCBS  survey. 
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Table  26.  Comparison  of  Whether  Beneficiary  Kept  the  Handbook 


Survey 

Item  # 

Question 

Response  Analyzed 

Frequency 

MCBS 

BK  40 

Do  you  still  have  (it)? 

Yes 

89% 

M&Y  2000 

Treat.78 

Did  you  keep  (it)? 

Yes 

99% 

Likewise,  a  greater  percentage  of  respondents  in  the  Medicare  &  You  survey  indicated  that  they 
still  had  the  handbook  when  compared  to  the  MCBS  sample  respondents. 


Four  issues  affect  a  straight-forward  comparison  of  these  results: 

■  The  sample  populations  are  different.  The  MCBS  sample  analyzed  above  is  only  for 
five  states  in  1 998  which  had  received  the  handbook,  whereas  the  Medicare  &  You 
sample  is  a  nationally  representative  sample.  Differences  in  the  samples  may  have 
contributed  to  the  difference  in  the  responses. 

■  The  version  of  the  handbook  differed  between  samples.  The  MCBS  sample  received 
the  1998  version  of  the  handbook  while  the  Medicare  &  You  sample  received  the 
1999  version  of  the  handbook.  Differences  in  the  handbook  itself  may  have 
contributed  in  part  to  whether  sample  persons  read  and  kept  their  handbook. 

■  The  Medicare  &  You  results  may  be  higher  because  the  sampled  population  was 
participating  in  the  evaluation.  As  a  result,  they  may  have  been  sensitized  to  the 
experiment  and  wanted  to  "pass  the  test"  concerning  the  handbook,  which  may  have 
contributed  to  the  differences  in  response. 

-    The  timeframes  of  each  survey  are  different.  The  MCBS  Round  23  was  fielded  in 
early  1999  while  the  Medicare  &  You  evaluation  was  fielded  in  November  1999  - 
March  2000.  Temporal  differences  may  in  part  have  affected  whether  sample  persons 
read  and  kept  their  handbook. 

6.2.2  Comparison  of  8  Quiz  Questions 

Rounds  23  and  24  of  the  MCBS  and  the  survey  for  the  national  evaluation  of  the 
Medicare  &  You  handbook  also  contained  8  similar  questions  on  a  variety  of  issues  pertaining  to 
knowledge  of  the  Medicare  program  and  actions  beneficiaries  have  taken.  Five  of  these  eight 
questions  were  exactly  the  same  in  the  MCBS  and  the  Medicare  &  You  evaluation  and  are 
shown  in  Table  27  below  with  weighted  frequencies  for  each  question  in  both  surveys. 
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Table  27.  Comparison  of  Responses  to  the  1998  MCBS  (Round  23)  and  the  Medicare 
«&You  Evaluation  Survey  Questions 


MCBS# 

M&Y# 

Question/Statement 

Response  Analyzed 

MCBS  % 

M&Y  % 

BK  46 

Treat.  9 

The  Medicare  program  has  recently 
begun  to  offer  more  information  and 
help  in  order  to  answer  your  Medicare 
questions.  (True/False) 

True/Correct 

45.9 

92.2 

BK  11 

Treat  31 

In  the  past  year,  have  you  tried  to  find 
information  about  what  medical  services 
Medicare  covers  and  does  not  cover? 

Yes 

8.0 

31.6 

BK2 

Treat.  30 

How  satisfied  are  you  in  general  with  the 
availability  of  information  about  the 
Medicare  program  when  you  need  it? 

Persons  responding 
"Very  satisfied"  and 
"Satisfied" 

93.0 

79.1 

BK  1 

Treat.  32 

How  much  do  you  think  you  know  about 
the  Medicare  program? 

Persons  responding 
"just  about  everything," 
"most,"  and  "some"  of 
what  they  need  to  know 

63.4 

67.1 

BN  1 

Treat.  33 

How  much  do  you  feel  you  know  about 
what  medical  services  Medicare  covers 
or  does  not  cover? 

Persons  responding 
"just  about  everything," 
"most,"  and  "some"  of 
what  they  need  to  know 

70.7 

68.0 

BN4 

Treat.  35 

How  much  do  you  feel  you  know  about 
the  availability  and  benefits  of  . . 
. .  .Medicare  managed  care  plans 
(Remainder  in  MCBS) 
. .  .other  health  plan  choices  available 
under  Medicare  (Remainder  in  M&  Y) 

Persons  responding 
"just  about  everything," 
"most,"  and  "some"  of 
what  they  need  to  know 

41.2 

46.2 

Responses  to  the  first  three  questions  in  the  table  (MCBS  BK  46,  1 1,  and  2)  differed 
markedly  between  the  two  populations.  In  the  first  question,  substantially  more  people  in  the 
Medicare  &  You  sample  responded  that  there  was  more  information  available  in  the  Medicare 
program  to  answer  questions.  This  difference  in  part  may  be  attributed  to  respondents' 
participation  in  the  evaluation  study. 


In  the  second  question  above  (MCBS  BK  1 1),  a  larger  percentage  of  people  in  the 
Medicare  &  You  sample  tried  to  find  information  about  what  medical  services  Medicare  covers 
and  does  not  cover.  Again,  Medicare  &  You  respondents  may  have  been  tempted  to  use  the 
handbook  to  find  information  about  certain  issues  because  of  their  participation  in  a  study.  In  the 
third  question  (MCBS  BK  2),  more  persons  in  the  MCBS  sample  were  satisfied  with  the 
availability  of  information  about  the  Medicare  program  when  they  needed  it.  It  is  difficult  to 
determine  why  the  differences  are  so  large  between  the  samples  on  this  question. 

In  the  fourth  question  (MCBS  BKl)  and  the  fifth  question  (MCBS  BN  1)  in  the  table, 
roughly  the  same  percentage  of  people  in  the  two  samples  answered  they  knew  just  about 
everything,  most,  or  some  of  what  they  needed  to  know.  These  questions  about  perceived 
knowledge  of  the  Medicare  program  and  the  services  it  covers  may  serve  as  proxies  for  the 
amoimt  of  knowledge  these  persons  have. 
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A  sixth  question  (MCBS  BN  4)  asked  respondents  about  their  knowledge  of  other  health 
plan  choices  in  Medicare  but  with  slightly  different  language.  Approximately  41 .2%  of 
respondents  in  the  MCBS  sample  responded  "just  about  everything,  most,  and  some"  of  what 
they  needed  to  know,  whereas  46.2%  of  respondents  in  the  Medicare  &You  sample  reported  the 
same,  not  a  very  substantial  difference  in  terms  of  perceived  knowledge. 

The  responses  from  the  remaining  two  (out  of  the  eight  questions  that  were  similar 
between  the  surveys)  could  not  be  compared  directly.  A  general  question  about 
understandability  of  the  Medicare  program  was  exactly  the  same  in  the  MCBS  and  the  Medicare 
&  You  surveys,  but  the  question  has  not  been  asked  in  the  MCBS  since  1997.  In  addition, 
psychometric  analyses  performed  on  this  measure  in  Section  3.2  of  this  report  foimd  that  the 
question  was  not  correlated  with  the  other  knowledge  indices  tested,  so  it  may  be  measuring  a 
somewhat  different  construct  than  the  other  three  scales  analyzed  which  measured  knowledge. 
Given  the  large  temporal  difference  and  the  psychometric  analysis,  comparisons  should  not  be 
made  using  this  question. 

Question  BK  15  in  the  MCBS  asks  respondents  if  they  "tried  to  find  out  about  the 
availability  and  benefits  of  Medicare  managed  care  plans,  such  as  HMOs."  A  similar,  but  not 
exactly  the  same  question  from  the  Medicare  &  You  survey  asks  respondents  how  much  they 
"feel  they  know  about  the  availability  and  benefits  of  HMOs."  While  the  topic  is  the  same,  the 
MCBS  question  asks  about  an  action,  whereas  the  Medicare  &  You  question  asks  about 
perceived  knowledge.  Given  this  difference,  the  two  questions  should  not  be  compared. 

7.0    Conclusions  and  Policy  Recommendations 

Two  knowledge  measures  were  identified  to  have  sufficient  internal  consistency 
reliability  and  construct  validity  to  warrant  use  as  baseline  measures — the  Medicare  Knowledge 
index  (the  4-item  quiz)  contained  in  the  1996  MCBS  and  the  Perceived  Medicare  Knowledge 
index  found  in  the  1996  and  1998  MCBS.  Other  measures  investigated  and  tested  were  not 
sufficiently  reliable  or  valid. 

Chi-square  analyses  were  performed  on  both  of  these  knowledge  measures  using  16 
MCBS  beneficiary  characteristics  to  determine  if  there  were  differences  in  the  distribution  of 
knowledge  for  any  given  characteristic.  For  many  beneficiary  characteristics,  there  was  little 
meaningful  variance  in  the  distribution  although  the  tests  were  significant.  Changes  to  the 
NMEP  as  currently  implemented  may  be  beneficial  when  there  is  a  larger  variance  in  responses, 
and  the  changes  are  targeted  at  those  individuals  who  scored  much  lower  than  their  counterparts. 
Selective  targeting  of  any  given  suggested  change  in  NMEP  implementation  would  be  more  cost 
effective  than  sending  the  intervention  to  the  entire  population. 

Based  on  the  results  of  the  scale-level  and  item-level  analyses,  the  knowledge  index  from 
National  Evaluation  of  the  Medicare  &  You  2000  Handbook  performed  better  than  the 
knowledge  questions  from  the  MCBS.  The  National  Evaluation  index  met  several  of  the  criteria 
evaluated  in  this  report.  The  index  had  the  highest  reliability  of  the  three  scales  and 
demonstrated  evidence  of  construct  validity.  The  results  of  the  factor  analyses  as  well  as  the 
content  of  the  items  suggest  that  this  scale  measures  a  wider  range  of  topics  in  comparison  to 
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other  knowledge  evaluated,  in  part  because  it  has  more  questions.  This  diversity  allows  the  scale 
to  more  accurately  assess  the  knowledge  beneficiaries  need  to  navigate  the  Medicare  program 
successfully.  The  MCBS  8-item  quiz  and  the  Kansas  City  15-item  knowledge  index  also  contain 
potential  candidate  items  for  inclusion  in  future  rounds  of  the  MCBS.  Questions  used  in  the 
MCBS  to  evaluate  knowledge  should  directly  reflect  the  content  of  the  version  of  the  Medicare 
&  You  handbook  distributed  to  beneficiaries  just  prior  to  fielding  the  MCBS.  As  the  content  and 
focus  of  Medicare  &  You  changes,  so  too  must  the  knowledge  index  questions. 

On  questions  that  were  found  in  both  the  MCBS  and  the  National  Evaluation  of  the 
Medicare  &  You  handbook,  participants  in  the  Medicare  &  You  sample  may  have  scored  more 
favorably  in  general  on  various  questions  because  they  knew  they  were  participating  in  the 
national  evaluation. 

8.0  Recommendations  for  Future  Research 

8.1  Methods  for  Conducting  Longitudinal  Analyses 

Future  work  is  needed  to  develop  longitudinal  analyses  to  more  accurately  monitor  the 
effect  of  the  NMEP  in  the  same  population  as  opposed  to  comparing  the  effects  across  different 
samples  using  cross  sectional  analyses.  Longitudinal  analyses  focused  on  the  MCBS  population 
will  be  beneficial  particularly  beginning  with  the  1999  and  2000  MCBS  rounds.  Several 
technical  issues  concerning  a  methodology  to  be  employed  for  longitudinal  analyses  must  be 
resolved  before  the  MCBS  measures  can  be  used  to  provide  useful  policy  information  for  the 
NMEP  over  time.  In  part  these  issues  include: 

•  whether  data  should  be  pooled  for  measuring  certain  outcomes  over  time, 

•  how  predictions  should  be  made  for  changes  in  beneficiary  attributes,  and 

•  what  types  of  statistical  tests  should  be  used  for  measuring  certain  outcomes  over 
time. 

8.2  Potential  Measures  of  the  Impact  of  the  NMEP 

The  goals  of  the  NMEP  include  (a)  Access,  (b)  Awareness,  (c)  Understanding,  and  (d) 
Use/Impact/Trust.  The  knowledge  indices  evaluated  in  this  report  address  only  the  first  three 
goals.  Although  an  increase  in  understanding  (knowledge)  might  be  thought  of  as  an  "impact"  of 
the  NMEP,  it  is  an  intermediate  prerequisite  to  the  improvement  in  other  outcomes.  Future 
studies  should  investigate  possible  measures  of  knowledge  use,  confidence  in  one's  ability  to 
make  the  right  choices  or  to  understand  the  system,  and  trust.  For  example,  the  MCBS  Perceived 
Medicare  Knowledge  index  may  be  appropriate  as  a  measure  of  the  impact  of  the  NMEP.  This 
index  allows  respondents  to  express  how  much  they  feel  they  know  about  a  particular  topic  and 
may  be  useful  for  evaluating  whether  the  NMEP  has  an  effect  on  beneficiaries'  confidence  in 
their  knowledge  of  the  Medicare  program.  In  addition,  we  found  the  Perceived  Medicare 
Knowledge  index  to  have  good  psychometric  properties  (see  Bann  et  al,  2000). 
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8.3  Direct  Comparisons  Among  Knowledge  Questions 

In  Section  6  we  evaluated  the  questions  from  three  different  knowledge  indices:  the 
MCBS  8-item  quiz,  the  National  Evaluation  index,  and  the  Kansas  City  index.  However, 
because  the  knowledge  indices  were  administered  to  different  groups  of  respondents,  we  were 
unable  to  make  direct  comparisons  between  the  items.  This  prevented  us  from  determining 
whether  differences  in  the  item  parameters  were  due  to  differences  in  the  items  or  differences  in 
the  respondents.  For  example,  if  a  larger  percentage  of  respondents  got  high  knowledge  scores 
in  the  Kansas  City  than  in  the  National  Evaluation  data  sets,  we  cannot  know  whether  this  is 
because  the  beneficiaries  who  responded  to  the  National  Evaluation  survey  were  less  informed  or 
whether  they  received  more  difficult  items.  To  more  effectively  compare  the  three  sets  of 
knowledge  questions,  all  of  the  items  should  be  administered  to  the  same  group  of  respondents  in 
future  data  collections.  The  psychometric  properties  of  the  items  could  then  be  evaluated  and 
compared  to  determine  which  items  most  effectively  measure  knowledge  and  the  best 
performing  items  could  then  be  selected  and  assembled  into  a  new,  more  effective  knowledge 
index. 

8.4  Development  of  a  Knowledge  Index  Using  Item  Response  Theory 

Traditionally,  questions  on  the  Medicare  Current  Beneficiary  Survey  (MCBS)  have 
changed  from  year  to  year  to  address  the  newest  features  of  Medicare  health  plans  and  to  adapt 
to  the  changing  priorities  and  goals  of  HCFA.  However,  the  changing  content  makes  it  difficult 
to  measure  improvement  or  decline  in  beneficiary  knowledge  from  year  to  year  and  therefore  to 
evaluate  the  effectiveness  of  interventions  designed  to  increase  beneficiary  knowledge.  For 
example,  due  to  the  changing  content,  we  found  that  the  MCBS  could  provide  only  a  limited 
amount  of  information  on  the  effectiveness  of  the  National  Medicare  Education  Program 
(NMEP)  interventions.  Item  Response  Theory  (IRT)  can  be  used  to  remedy  this  problem  by 
assigning  a  comparable  metric  to  knowledge  item  sets  that  will  differ  from  year-to-year  in  future 
rounds  of  the  MCBS.  IRT  enables  the  creation  of  equivalent  scores  from  two  forms  of  a  scale 
that  contain  only  some  items  in  common  (Wainer,  1990). 

The  use  of  IRT  to  develop  and  evaluate  measures  of  knowledge  has  been  well  established 
in  the  field  of  educational  testing.  For  example,  many  large-scale  testing  programs,  such  as  the 
Law  School  Admission  Test  (LSAT)  and  Graduate  Record  Examination  (GRE),  use  IRT  to 
equate  different  test  forms.  Using  IRT  allows  these  educational  testing  programs  to  change  test 
items  frequently  to  prevent  item  memorization  while  still  assuring  that  respondents  receiving 
different  forms  have  comparable  test  scores.  Increasingly,  IRT  methods  are  also  being  applied  to 
health  services  research.  For  example,  on  September  16  and  17,  1999,  the  Agency  of  Healthcare 
Research  and  Quality  and  the  National  Cancer  Institute  hosted  an  invitation-only  health 
outcomes  measures  methodology  symposium  for  which  seven  papers  were  commissioned.  Two 
of  these  papers  addressed  the  use  of  Item  Response  Theory  to  develop  and  evaluate  health 
measures. 

The  process  for  developing  the  MCBS  knowledge  index  via  IRT  would  consist  of  three 
steps:  (1)  establishment  of  content  categories,  (2)  development  of  a  "pool"  of  candidate  items, 
and  (3)  pre-testing  of  the  items.  The  first  step  would  be  to  decide  which  topics  or  content 
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categories  should  be  included  in  the  knowledge  scale,  as  well  as  the  proportion  of  items  that 
should  address  a  particular  topic.  These  content  categories  should  cover  both  core  knowledge 
areas  that  generally  remain  consistent  from  year  to  year  as  well  as  supplemental  topics  that  may 
change  more  frequently.  One  possible  set  of  content  categories  could  be  the  seven  topic  areas 
listed  in  the  National  Medicare  &  You  evaluation  report:  (1)  awareness  of  Medicare  options,  (2) 
access  to  traditional  Medicare,  (3)  cost  implications  of  insurance  choices,  (4)  coverage/benefits, 
(5)  plan  rules/restrictions,  (6)  availability  of  information,  and  (7)  beneficiary  rights  (McCormack, 
Ross,  Daugherty,  &  Garfinkel,  2000). 

After  finalizing  the  content  categories,  the  next  step  would  be  to  develop  a  pool  of  items 
that  address  the  relevant  topics.  Possible  sources  of  these  items  may  be  previous  rounds  of  the 
MCBS,  the  Kansas  City  Medicare  &  You  evaluation,  or  the  National  Medicare  &  You 
evaluation.  If  needed,  additional  items  would  also  be  developed  to  assure  that  there  was  a  large 
pool  of  items  covering  each  of  the  content  categories.  If  new  items  were  created,  they  would 
also  need  to  be  cognitively  tested  on  a  small  sample  of  Medicare  beneficiaries.  This  technique 
can  detect  potential  problems  with  respondent  understanding  or  interpretation  of  the  items  that 
may  not  be  uncovered  by  other  review  processes  (Forsyth  &  Lessler,  1991).  If  problems  were 
discovered,  items  would  be  modified  after  the  interviews. 

Finally,  a  pretest  would  be  conducted  in  which  the  knowledge  questions  would  be 
administered  to  a  sample  of  Medicare  beneficiaries.  Data  from  the  pre-test  would  be  analyzed 
using  IRT  methods  to  select  the  items  for  the  final  knowledge  scale.  The  IRT  analyses  would 
provide  information  on  the  psychometric  properties  of  the  questions,  such  as  the  level  of 
difficulty  and  discrimination,  as  well  as  detect  possible  problems  with  the  items  that  may  not  be 
uncovered  using  traditional  item  analyses.  Items  would  be  selected  if  they  demonstrate  high 
levels  of  discrimination  and  low  levels  of  guessing.  Also,  items  with  varying  levels  of  difficulty 
would  be  chosen  to  assure  that  the  scale  would  be  appropriate  for  beneficiaries  with  different 
levels  of  knowledge. 

After  the  knowledge  measure  had  been  developed,  it  could  be  adapted  in  future  years  to 
fit  possible  changes  in  the  Medicare  program  by  replacing  outdated  items  with  new  items  that 
address  these  changes.  Because  IRT  can  produce  comparable  scores  from  different  forms  of  a 
scale  when  the  two  forms  contain  common  items,  the  scores  from  the  different  years  could  then 
be  used  to  evaluate  changes  in  knowledge  over  time. 

A  knowledge  index  developed  using  this  technique  would  provide  many  benefits.  First, 
the  knowledge  scale  would  be  flexible,  allowing  a  proportion  of  items  to  be  replaced  or  changed. 
Therefore,  it  could  be  adapted  to  reflect  new  aspects  of  the  Medicare  program  as  well  as  changes 
in  existing  features.  At  the  same  time,  this  knowledge  index  would  allow  HCFA  to  monitor  the 
knowledge  of  Medicare  beneficiaries  and  determine  whether  knowledge  levels  increase  or 
decrease.  This  information  could  then  be  used  to  determine  when  it  is  necessary  to  create  new 
educational  efforts  and  to  evaluate  the  effect  of  these  efforts  once  they  were  implemented. 
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